{"id":9200,"date":"2022-07-25T15:57:14","date_gmt":"2022-07-25T20:57:14","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9200"},"modified":"2022-07-25T16:09:46","modified_gmt":"2022-07-25T21:09:46","slug":"logstash-filtering-data-with-ruby","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9200","title":{"rendered":"Logstash – Filtering data with Ruby"},"content":{"rendered":"\n

I’ve been working on forking log data into two different indices based on an element contained within the record — if the filename being sent includes the string “BASELINE”, then the data goes into the baseline index, otherwise it goes into the scan index. The data being ingested has the file name in “@fields.myfilename”<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

It took a while to figure out how to get the value from the current data — event.get(‘[@fields][myfilename]’) to get the @fields.myfilename value. <\/p>\n\n\n\n

The following logstash config accepts JSON inputs, parses the underscore-delimited filename into fields, replaces the dashes with underscores as KDL doesn’t handle dashes and wildcards in searches, and adds a flag to any record that should<\/em> be a baseline. In the output section, that flag is then used to publish data to the appropriate index based on the baseline flag value. <\/p>\n\n\n

\ninput {\n  tcp {\n    port => 5055\n    codec => json\n  }\n}\nfilter {\n        # Sample file name: scan_ABCDMIIWO0Y_1-A-5-L2_BASELINE.json\n        ruby {  code => "\n                        strfilename = event.get('[@fields][myfilename]')\n                        arrayfilebreakout = strfilename.split('_')\n                        event.set('hostname', arrayfilebreakout[1])\n                        event.set('direction',arrayfilebreakout[2])\n                        event.set('parseablehost', strfilename.gsub('-','_'))\n\n                        if strfilename.downcase =~ \/baseline\/\n                                event.set('baseline', 1)\n                        end" }\n}\noutput {\n        if [baseline] == 1 {\n                elasticsearch {\n                        action => "index"\n                        hosts => ["https:\/\/elastic.example.com:9200"]\n                        ssl => true\n                        cacert => ["\/path\/to\/logstash\/config\/certs\/My_Chain.pem"]\n                        ssl_certificate_verification => true\n                        # Credentials go here\n                        index => "ljr-baselines"\n                }\n        }\n        else{\n              elasticsearch {\n                        action => "index"\n                        hosts => ["https:\/\/elastic.example.com:9200"]\n                        ssl => true\n                        cacert => ["\/path\/to\/logstash\/config\/certs\/My_Chain.pem"]\n                        ssl_certificate_verification => true\n                        # Credentials go here\n                        index => "ljr-scans-%{+YYYY.MM.dd}"\n                }\n        }\n}\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"
I’ve been working on forking log data into two different indices based on an element contained within the record — if the filename being sent includes the string “BASELINE”, then the data goes into the baseline index, otherwise it goes into the scan index. The data being ingested has the file name in “@fields.myfilename” It …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1643,1681],"class_list":["post-9200","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-logstash","tag-ruby"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9200","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9200"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9200\/revisions"}],"predecessor-version":[{"id":9205,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9200\/revisions\/9205"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9200"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9200"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9200"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}