{"id":9097,"date":"2022-06-24T13:08:42","date_gmt":"2022-06-24T18:08:42","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9097"},"modified":"2022-06-24T13:09:10","modified_gmt":"2022-06-24T18:09:10","slug":"9097","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9097","title":{"rendered":"Logstash"},"content":{"rendered":"

General Info<\/strong><\/p>\n

Logstash is a pipeline based data processing service. Data comes into logstash, is manipulated, and is sent elsewhere. The source is maintained on GitHub by ElasticCo<\/a>.<\/p>\n

Installation<\/strong><\/p>\n

Logstash<\/a> was downloaded from ElasticCo and installed from a gzipped tar archive to the \/opt\/elk\/logstash folder.<\/p>\n

Configuration<\/strong><\/p>\n

The Logstash server is configured using the logstash.yml file<\/a>.<\/p>\n

Logstash uses Log4J 2 for logging. Logging configuration<\/a> is maintained in the log4j2.properties file<\/p>\n

Logstash is java-based, and the JVM settings are maintained in the jvm.options file \u2013 this includes min heap space, garbage collection configuration, JRuby settings, etc.<\/p>\n

Logstash loads the pipelines defined in \/opt\/elk\/logstash\/config\/pipelines.yml \u2013 each pipeline needs an ID and a path to its configuration. The path can be to a config file or to a folder of config files for the pipeline. The number of workers for the pipeline defaults to the number of CPUs, so we normally define a worker count as well \u2013 this can be increased as load dictates.<\/p>\n

– pipeline.id: LJR
\npipeline.workers: 2
\npath.config: “\/opt\/elk\/logstash\/config\/ljr.conf”<\/p>\n

Each pipeline is configured in an individual config file<\/a> that defines the input, any data manipulation to be performed, and the output.<\/p>\n

Testing Configuration<\/strong><\/p>\n

As we have it configured, you must reload Logstash to implement any configuration changes. As errors in pipeline definitions will prevent the pipeline from loading, it is best to test the configuration prior to restarting Logstash.<\/p>\n

\/opt\/elk\/logstash\/bin\/logstash –config.test_and_exit -f ljr_firewall_logs_wip.conf<\/p>\n

Some errors may occur \u2013 if the test ends with “Configuration OK”, then it’s OK!<\/p>\n

<\/p>\n

Automatic Config Reload<\/strong><\/p>\n

The configuration can<\/em> automatically be reloaded<\/a> when changes to config files are detected. This doesn’t give you the opportunity to test a configuration prior to it going live on the server (once it’s saved, it will be loaded … or not<\/em> loaded if there’s an error)<\/p>\n

Input<\/strong><\/p>\n

Input instructs logstash about what format data the pipeline will receive \u2013 is JSON data being sent to the pipeline, is syslog sending log data to the pipeline, or does data come from STDIN? The types of data that can be received are defined by the input plugins<\/a>. Each input has its own configuration parameters. We use Beats<\/a>, syslog<\/a>, JSON (a codec, not a filter type), Kafka<\/a>, stuff<\/p>\n

The input configuration also indicates which port to use for the pipeline \u2013 this needs to be unique!<\/p>\n

Input for a pipeline on port 5055 receiving JSON formatted data<\/p>\n

<\/p>\n

Input for a pipeline on port 5100 (both TCP and UDP) receiving syslog data<\/p>\n

<\/p>\n

Output<\/strong><\/p>\n

Output is similarly simple \u2013 various output plugins<\/a> define the systems to which data can be shipped. Each output has its own configuration parameters \u2013 ElasticSearch<\/a>, Kafka<\/a>, and file <\/a>are the three output plug-ins we currently use.<\/p>\n

ElasticSearch<\/strong><\/p>\n

Most of the data we ingest into logstash is processed and sent to ElasticSearch. The data is indexed and available to users through ES and Kibana.<\/p>\n

Kafka<\/strong><\/p>\n

Some data is sent to Kafka basically as a holding queue. It is then picked up by the “aggregation” logstash server, processed some more, and relayed to the ElasticSearch system.<\/p>\n

File<\/strong><\/p>\n

File output is generally used for debugging \u2013 seeing the output data allows you to verify your data manipulations are working property (as well as just make sure you see data transiting the pipeline without resorting to tcpdump!).<\/p>\n

Filter<\/strong><\/p>\n

Filtering allows data to be removed, attributes to be added to records, and parses data into fields. The types of filters that can be applied are defined by the filter plugins<\/a>. Each plugin has its own documentation. Most of our data streams are filtered using Grok \u2013 see below for more details on that.<\/p>\n

Conditional rules<\/a> can be used in filters. This example filters out messages that contain the string “FIREWALL”, “id=firewall”, or “FIREWALL_VRF” as the business need does not require these messages, so there’s no reason to waste disk space and I\/O processing, indexing, and storing these messages.<\/p>\n

<\/p>\n

This example adds a field, ‘sourcetype’, with a value that is based on the log file path.<\/p>\n

<\/p>\n

Grok<\/strong><\/p>\n

The grok filter<\/a> is a Logstash plugin that is used to extract data from log records \u2013 this allows us to pull important information into distinct fields within the ElasticSearch record. Instead of having the full message in the ‘message’ field, you can have success\/failure in its own field, the logon user in its own field, or the source IP in its own field. This allows more robust reporting. If the use case just<\/em> wants to store data, parsing the record may not be required. But, if they want to report on the number of users logged in per hour or how much data is sent to each IP address, we need to have the relevant fields available in the document.<\/p>\n

Patterns used by the grok filter are maintained in a Git repository<\/a> \u2013 the grok-patterns contains the data types like ‘DATA’ in %{DATA:fieldname}<\/p>\n

The following are the ones I’ve used most frequently:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Name<\/strong><\/td>\nField Type<\/strong><\/td>\nPattern Notes<\/strong><\/td>\nNotes<\/strong><\/td>\n<\/tr>\n
DATA<\/td>\nText data<\/td>\n.*?<\/td>\nThis does not expand to the most matching characters \u2013 so looking for foo.*?bar in “foobar<\/strong> is not really a word, but foobar gets used a lot in IT documentation” will only match the bold portion of the text<\/td>\n<\/tr>\n
GREEDYDATA<\/td>\nText data<\/td>\n.*<\/td>\nWhereas this matches the most<\/em> matching characters \u2013 so foo.*bar in “foobar is not really a word, but foobar<\/strong> gets used a lot in IT documentation” matches the whole bold portion of the text<\/td>\n<\/tr>\n
IPV4<\/td>\nIPv4 address<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
IPV6<\/td>\nIPv6 address<\/td>\n<\/td>\n<\/td>\n<\/tr>\n
IP<\/td>\nIP address \u2013 either v4 or v6<\/td>\n(?:%{IPV6}|%{IPV4})<\/td>\nThis provides some flexibility as groups move to IPv6 \u2013 but it’s a more complex filter, so I’ve been using IPV4 with the understanding that we may need to adjust some parsing rules in the future<\/td>\n<\/tr>\n
LOGLEVEL<\/td>\nText data<\/td>\n<\/td>\nRegex to match list of standard log level strings \u2013 provides data validation over using DATA (i.e. if someone sets their log level to “superawful”, it won’t match)<\/td>\n<\/tr>\n
SYSLOGBASE<\/td>\nText data<\/td>\n<\/td>\nThis matches the standard start of a syslog record. Often used as “%{SYSLOGBASE} %{GREEDYDATA:msgtext}” to parse out the timestamp, facility, host, and program \u2013 the remainder of the text is mapped to “msgtext”<\/td>\n<\/tr>\n
URI<\/td>\nText data<\/td>\n<\/td>\nprotocol:\/\/stuff text is parsed into the protocol, user, host, path, and query parameters<\/td>\n<\/tr>\n
INT<\/td>\nNumeric data<\/td>\n(?:[+-]?(?:[0-9]+))<\/td>\nSigned or unsigned integer<\/td>\n<\/tr>\n
NUMBER<\/td>\nNumeric data<\/td>\n<\/td>\nCan include a casting like\u00a0%{NUMBER:fieldname;int} or %{NUMBER:fieldname;float}<\/td>\n<\/tr>\n
TIMESTAMP_ISO8601<\/td>\nDateTime<\/td>\n%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?<\/td>\nThere are various other date patterns depending on how the string will be formatted. This is the one that matches YYYYMMDDThh:mm:ss<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Parsing an entire log string<\/strong><\/p>\n

In a system with a set format for log data, parsing the entire line is reasonable \u2013 and, often, there will be a filter for well-known log types. I.E. if you are using the default Apache HTTPD log format, you don’t need to write a filter for each component of the log line\u00a0\u2013 just match either the HTTPD_COMBINEDLOG or HTTPD_COMMONLOG pattern.<\/p>\n

match => { “message” => “%{HTTPD_COMMONLOG}” }<\/p>\n

But you can create your own filter as well \u2013 internally developed applications and less common vendor applications won’t have prebuilt filter rules.<\/p>\n

match => { “message” => “%{TIMESTAMP_ISO8601:logtime} – %{IPV4:srcip} – %{IPV4:dstip} – %{DATA:result}” }<\/p>\n

Extracting an array of data<\/strong><\/p>\n

Instead of trying to map an entire line at once, you can extract individual data elements by matching an array of patterns within the message.<\/p>\n

match => { “message” => [“srcip=%{IPV4:src_ip}”
\n, “srcport=%{NUMBER:srcport:int}”
\n,”dstip=%{IPV4:dst_ip}”
\n,”dstport=%{NUMBER:dstport:int}”] }<\/p>\n

This means the IP and port information will be extracted regardless of the order in which the fields are written in the log record. This also allows you to parse data out of log records where multiple different formats are used (as an example, the NSS Firewall logs) instead of trying to write different parsers for each of the possible string combinations.<\/p>\n

Logstash, by default, breaks when a match is found. This means you can ‘stack’ different filters instead of using if tests. Sometimes, though, you don’t want<\/em> to break when a match is found\u00a0\u2013 maybe you are extracting a bit of data that gets used in another match. In these cases, you can set break_on_match to ‘false’ in the grok rule.<\/p>\n

I have also had to set break_on_match when extracting an array of values from a message.<\/p>\n

<\/p>\n

Troubleshooting<\/strong><\/p>\n

Log Files<\/strong><\/p>\n

Logstash logs output to \/opt\/elk\/logstash\/logs\/logstash-plain.log\u00a0\u2013 the logging level is defined in the \/opt\/elk\/logstash\/config\/log4j2.properties configuration file.<\/p>\n

Viewing Data Transmitted to a Pipeline<\/strong><\/p>\n

There are several ways to confirm that data is being received by a pipeline \u2013 tcpdump can be used to verify information is being received on the port. If no data is being received, the port may be offline (if there is an error in the pipeline config, the pipeline will not load \u2013 grep \/opt\/elk\/logstash\/logs\/logstash-plain.log for the pipeline name to view errors), there may be a firewall preventing communication, or the sender could not be transmitting data.<\/p>\n

tcpdump dst port 5100 -vv<\/p>\n

If data is confirmed to be coming into the pipeline port, add a “file” output filter to the pipeline.<\/p>\n

Issues<\/strong><\/p>\n

Data from filebeat servers not received in ElasticSearch<\/strong><\/p>\n

We have encountered a scenario were data from the filebeat servers was not being transmitted to ElasticSearch. Monitoring the filebeat server did not show any data being sent. Restarting the Logstash servers allowed data to be transmitted as expected.<\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

General Info Logstash is a pipeline based data processing service. Data comes into logstash, is manipulated, and is sent elsewhere. The source is maintained on GitHub by ElasticCo. Installation Logstash was downloaded from ElasticCo and installed from a gzipped tar archive to the \/opt\/elk\/logstash folder. Configuration The Logstash server is configured using the logstash.yml file. …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1589,1643],"class_list":["post-9097","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elk","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9097","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9097"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9097\/revisions"}],"predecessor-version":[{"id":9105,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9097\/revisions\/9105"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9097"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9097"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9097"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}