{"id":9041,"date":"2022-06-02T20:57:12","date_gmt":"2022-06-03T01:57:12","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9041"},"modified":"2022-06-09T10:38:18","modified_gmt":"2022-06-09T15:38:18","slug":"using-logstash-to-send-data-to-elasticsearch","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9041","title":{"rendered":"Using Logstash to Send Data to ElasticSearch"},"content":{"rendered":"

Create a logstash pipeline<\/p>\n

    \n
  1. The quickest thing to do is copy the config of a similar use case and adjusted the pipeline port (and adjusted the ES destination index). But, if this is a unique scenario, build a new pipeline configuration. I am creating a TCP listener that receives data from Python using the python-logstash module. In this configuration, logstash will create the index as needed with YYYY-MM-dd appended to the base index name.
    \n\"Text\n\nDescription<\/li>\n
  2. Edit the pipelines.yml to register the config you just created
    \n<\/li>\n
  3. Restart logstash to activate the new pipeline<\/li>\n
  4. Use netstat -nap | grep `pidof java` to ensure the server is listening on the new port
    \n<\/li>\n
  5. Add the port to the runtime firewalld rules and test that the port is functional (firewall-cmd –zone=public –add-port=5055\/tcp)<\/li>\n
  6. Assuming the runtime rule has not had any unexpected results, register a permanent firewalld rule (firewall-cmd –permanent –zone=public –add-port=5055\/tcp)<\/li>\n<\/ol>\n

    We now have a logstash data collector ready. We next need to create the index templates in ES<\/p>\n

      \n
    1. Log into Kibana<\/li>\n
    2. Create an ILM policy \u2013 this policy rolls indices into the warm phase after 2 days and forces merge. It also deletes records after 20 days.
      \n{ “policy”: { “phases”: { “hot”: { “min_age”: “0ms”, “actions”: { “set_priority”: { “priority”: 100 } } }, “warm”: { “min_age”: “2d”, “actions”: { “forcemerge”: { “max_num_segments”: 1 }, “set_priority”: { “priority”: 50 } } }, “delete”: { “min_age”: “20d”, “actions”: { “delete”: {} } } } } }<\/li>\n
    3. Create an index template — define the number of replicas<\/li>\n
    4. Send data through the pipeline \u2013 the index will get created per the template definitions and document(s) added to the index<\/li>\n<\/ol>\n

       <\/p>\n","protected":false},"excerpt":{"rendered":"

      Create a logstash pipeline The quickest thing to do is copy the config of a similar use case and adjusted the pipeline port (and adjusted the ES destination index). But, if this is a unique scenario, build a new pipeline configuration. I am creating a TCP listener that receives data from Python using the python-logstash …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1643],"class_list":["post-9041","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9041"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9041\/revisions"}],"predecessor-version":[{"id":9045,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9041\/revisions\/9045"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}