{"id":9492,"date":"2022-09-19T20:56:35","date_gmt":"2022-09-20T01:56:35","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9492"},"modified":"2022-10-05T15:07:33","modified_gmt":"2022-10-05T20:07:33","slug":"logstash-filtering-null-terminated-messages","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9492","title":{"rendered":"Logstash &#8211; Filtering Null-Terminated Messages"},"content":{"rendered":"<p>I have a syslog message that contains a null terminated string: <code>\"syslog_message\":\"A10\\u0000\"<\/code> &#8212; these messages represent is-alive checks from a load balancer to the logstash servers. I would prefer not to have thousands of &#8220;the A10 checked &amp; said logstash is still there&#8221; filling up Elasticsearch.<\/p>\n<p>Unfortunately, the logstash configuration doesn&#8217;t recognize unicode escape sequences &#8230; and it&#8217;s not like I can literally type a NULL the way I could type a <span id=\"degree-symbol\" class=\"font3\">\u00b0<\/span> or \u00e8<\/p>\n<p>I&#8217;ve been able to filter out any messages that <em>start with<\/em> A10. Since our &#8220;real&#8221; messages start with timestamps, I shouldn&#8217;t be dropping any good data, but there&#8217;s always the possibility. Without any way to indicate a null character, the closest match is <em>any single character<\/em> &#8230; and I&#8217;ve decided not to worry about a possible log message that is simply A101 or A10$ until we encounter a system that would send such messages.<\/p>\n<pre>#if [message] == \"A10\\u0000\"{  -- doesn't work\r\n#if [message] == \"A10\\\\u0000\"{ -- doesn't work\r\n#if [message] == 'A10\\u0000'{  -- doesn't work\r\n#if [message] =~ \/^A10\/{       -- this isn't great because of false positives, although *these* messages all start with a timestamp so are unlikely to match\r\nif [message] =~ \"^A10.$\" {\r\n     drop { }\r\n}<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have a syslog message that contains a null terminated string: &#8220;syslog_message&#8221;:&#8221;A10\\u0000&#8243; &#8212; these messages represent is-alive checks from a load balancer to the logstash servers. I would prefer not to have thousands of &#8220;the A10 checked &amp; said logstash is still there&#8221; filling up Elasticsearch. Unfortunately, the logstash configuration doesn&#8217;t recognize unicode escape sequences &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1643],"class_list":["post-9492","post","type-post","status-publish","format-standard","hentry","category-elk","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9492","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9492"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9492\/revisions"}],"predecessor-version":[{"id":9493,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9492\/revisions\/9493"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9492"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9492"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9492"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}