{"id":9022,"date":"2022-05-24T15:08:30","date_gmt":"2022-05-24T20:08:30","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9022"},"modified":"2022-05-24T15:08:31","modified_gmt":"2022-05-24T20:08:31","slug":"elk-monitoring","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9022","title":{"rendered":"ELK Monitoring"},"content":{"rendered":"\n

We have a number of logstash servers gathering data from various filebeat sources. We’ve recently experienced a problem where the pipeline stops getting data for some<\/em> of those sources. Not all — and restarting the non-functional filebeat source sends data for ten minutes or so. We were able to rectify the immediate problem by restarting our logstash services (IT troubleshooting step #1 — we restarted all of the filebeats and, when that didn’t help, moved on to restarting the logstashes)<\/p>\n\n\n\n

But we need to have a way to ensure this isn’t happening — losing days of log data from some sources is really bad<\/em>. So I put together a Python script to verify there’s something<\/em> coming in from each of the filebeat sources. <\/p>\n\n\n\n

pip install elasticsearch==7.13.4<\/p>\n\n\n

\n#!\/usr\/bin\/env python3\n#-*- coding: utf-8 -*-\n# Disable warnings that not verifying SSL trust isn't a good idea\nimport requests\nrequests.packages.urllib3.disable_warnings()\n\nfrom elasticsearch import Elasticsearch\nimport time\n\n# Modules for email alerting\nimport smtplib\nfrom email.mime.multipart import MIMEMultipart\nfrom email.mime.text import MIMEText\n\n\n# Config variables\nstrSenderAddress = "devnull@example.com"\nstrRecipientAddress = "me@example.com"\nstrSMTPHostname = "mail.example.com"\niSMTPPort = 25\n\nlistSplunkRelayHosts = ['host293', 'host590', 'host591', 'host022', 'host014', 'host135']\niAgeThreashold = 3600 # Alert if last document is more than an hour old (3600 seconds)\n\nstrAlert = None\n\nfor strRelayHost in listSplunkRelayHosts:\n\tiCurrentUnixTimestamp = time.time()\n\telastic_client = Elasticsearch("https:\/\/elasticsearchhost.example.com:9200", http_auth=('rouser','r0pAs5w0rD'), verify_certs=False)\n\n\tquery_body = {\n\t\t"sort": {\n\t\t\t"@timestamp": {\n\t\t\t\t"order": "desc"\n\t\t\t}\n\t\t},\n\t\t"query": {\n\t\t\t"bool": {\n\t\t\t\t"must": {\n\t\t\t\t\t"term": {\n\t\t\t\t\t\t"host.hostname": strRelayHost\n\t\t\t\t\t}\n\t\t\t\t},\n\t\t\t\t"must_not": {\n\t\t\t\t\t"term": {\n\t\t\t\t\t\t"source": "\/var\/log\/messages"\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tresult = elastic_client.search(index="network_syslog*", body=query_body,size=1)\n\tall_hits = result['hits']['hits']\n\n\tiDocumentAge = None\n\tfor num, doc in enumerate(all_hits):\n\t\tiDocumentAge =  (  (iCurrentUnixTimestamp*1000) - doc.get('sort')[0]) \/ 1000.0\n\n\tif iDocumentAge is not None:\n\t\tif iDocumentAge > iAgeThreashold:\n\t\t\tif strAlert is None:\n\t\t\t\tstrAlert = f"<tr><td>{strRelayHost}<\/td><td>{iDocumentAge}<\/td><\/tr>"\n\t\t\telse:\n\t\t\t\tstrAlert = f"{strAlert}\\n<tr><td>{strRelayHost}<\/td><td>{iDocumentAge}<\/td><\/tr>\\n"\n\t\t\tprint(f"PROBLEM - For {strRelayHost}, document age is {iDocumentAge} second(s)")\n\t\telse:\n\t\t\tprint(f"GOOD - For {strRelayHost}, document age is {iDocumentAge} second(s)")\n\telse:\n\t\tprint(f"PROBLEM - For {strRelayHost}, no recent record found")\n\n\nif strAlert is not None:\n\tmsg = MIMEMultipart('alternative')\n\tmsg['Subject'] = "ELK Filebeat Alert"\n\tmsg['From'] = strSenderAddress\n\tmsg['To'] = strRecipientAddress\n\n\tstrHTMLMessage = f"<html><body><table><tr><th>Server<\/th><th>Document Age<\/th><\/tr>{strAlert}<\/table><\/body><\/html>"\n\tstrTextMessage = strAlert\n\n\tpart1 = MIMEText(strTextMessage, 'plain')\n\tpart2 = MIMEText(strHTMLMessage, 'html')\n\n\tmsg.attach(part1)\n\tmsg.attach(part2)\n\n\ts = smtplib.SMTP(strSMTPHostname)\n\ts.sendmail(strSenderAddress, strRecipientAddress, msg.as_string())\n\ts.quit()\n\n<\/pre><\/div>\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
We have a number of logstash servers gathering data from various filebeat sources. We’ve recently experienced a problem where the pipeline stops getting data for some of those sources. Not all — and restarting the non-functional filebeat source sends data for ten minutes or so. We were able to rectify the immediate problem by restarting …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1642,1643,1644,664,633],"class_list":["post-9022","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-filebeat","tag-logstash","tag-monitoring","tag-python","tag-script"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9022","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9022"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9022\/revisions"}],"predecessor-version":[{"id":9023,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9022\/revisions\/9023"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9022"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9022"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9022"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}