{"id":9046,"date":"2022-06-03T15:25:23","date_gmt":"2022-06-03T20:25:23","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9046"},"modified":"2022-06-09T10:38:30","modified_gmt":"2022-06-09T15:38:30","slug":"using-filebeat-to-send-data-to-elasticsearch","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9046","title":{"rendered":"Using FileBeat to Send Data to ElasticSearch via Logstash"},"content":{"rendered":"<p>Before sending data, you need a pipleline on logstash to accept the data. If you are using an existing pipeline, you just need the proper host and port for the pipeline to use in the Filebeat configuration. If you need a <a href=\"https:\/\/www.rushworth.us\/lisa\/?p=9041\" target=\"_blank\" rel=\"noopener\">new pipeline<\/a>, the input needs to be of type &#8216;beats&#8217;<\/p>\n<ul>\n<li>If you want to parse the log data out into fields, use a grok filter. Grok parser rules can be tested at <a href=\"https:\/\/www.javainuse.com\/grok\" target=\"_blank\" rel=\"noopener\">https:\/\/www.javainuse.com\/grok<\/a><\/li>\n<\/ul>\n<ol>\n<li style=\"list-style-type: none;\"><\/li>\n<\/ol>\n<pre># Sample Pipeline Config:\r\ninput {\r\n  beats\u00a0\u00a0 {\r\n    host =&gt; \"logstashserver.example.com\"\r\n    port =&gt; 5057\r\n    client_inactivity_timeout =&gt; \"3000\"\r\n  }\r\n}\r\n\r\nfilter {\r\n  grok{\r\n     match =&gt; {\"message\"=&gt;\"\\[%{TIMESTAMP_ISO8601:timestamp}] %{DATA:LOGLEVEL} \\[Log partition\\=%{DATA:LOGPARTITION}, dir\\=%{DATA:KAFKADIR}\\] %{DATA:MESSAGE} \\(%{DATA:LOGSOURCE}\\)\"}\r\n  }\r\n}\r\n\r\noutput {\r\n  elasticsearch {\r\n    action =&gt; \"index\"\r\n    hosts =&gt; [\"https:\/\/eshost.example.com:9200\"]\r\n    ssl =&gt; true\r\n    cacert =&gt; [\"\/path\/to\/certs\/CA_Chain.pem\"]\r\n    ssl_certificate_verification =&gt; true\r\n    user =&gt;\"us3r1d\"\r\n    password =&gt; \"p@s5w0rd\"\r\n    index =&gt; \"ljrkafka-%{+YYYY.MM.dd}\"\r\n  }\r\n}<\/pre>\n<p>&nbsp;<\/p>\n<p>Download the appropriate version from <a href=\"https:\/\/www.elastic.co\/downloads\/past-releases#filebeat\" target=\"_blank\" rel=\"noopener\">https:\/\/www.elastic.co\/downloads\/past-releases#filebeat<\/a> \u2013 I am currently using 7.17.4 as we have a few CentOS + servers.<\/p>\n<p>Install the package (<tt>rpm -ihv filebeat-7.17.4-x86_64.rpm<\/tt>) \u2013 the installation package places the configuration files in \/etc\/filebeat and the binaries and other &#8220;stuff&#8221; in \/usr\/share\/filebeat<\/p>\n<p>Edit \/etc\/filebeat\/filebeat.yml<\/p>\n<ul>\n<li style=\"list-style-type: none;\">\n<ul>\n<li>Add inputs for log paths you want to monitor (this <em>may<\/em> be done under the module config if using a module config instead)<\/li>\n<li>Add an output for Logstash to the appropriate port for your pipeline:<br \/>\noutput.logstash:<br \/>\nhosts: [&#8220;logstashhost.example.com:5055&#8221;]<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Run filebeat in debug mode from the command line and watch for success or failure.<br \/>\n<tt>filebeat -e -c \/etc\/filebeat\/filebeat.yml -d \"*\"<\/tt><\/p>\n<p>Assuming everything is running well, use <tt>systemctl start filebeat<\/tt> to run the service and <tt>systemctl enable filebeat<\/tt> to set it to launch on boot.<\/p>\n<p>Filebeats will attempt to parse the log data and send a JSON object to the LogStash server. When you view the record in Kibana, you should see any fields parsed out with your grok rule \u2013 in this case, we have KAFKADIR, LOGLEVEL, LOGPARTITION, LOGSOURCE, and MESSAGE fields.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"897\" height=\"558\" class=\"wp-image-9047\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/06\/word-image-2.png\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/06\/word-image-2.png 897w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/06\/word-image-2-300x187.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/06\/word-image-2-768x478.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/06\/word-image-2-750x467.png 750w\" sizes=\"auto, (max-width: 897px) 100vw, 897px\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Before sending data, you need a pipleline on logstash to accept the data. If you are using an existing pipeline, you just need the proper host and port for the pipeline to use in the Filebeat configuration. If you need a new pipeline, the input needs to be of type &#8216;beats&#8217; If you want to &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1642,1643],"class_list":["post-9046","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-filebeat","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9046","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9046"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9046\/revisions"}],"predecessor-version":[{"id":9063,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9046\/revisions\/9063"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9046"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9046"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9046"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}