{"id":9745,"date":"2023-01-26T16:15:08","date_gmt":"2023-01-26T21:15:08","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9745"},"modified":"2023-01-27T15:55:00","modified_gmt":"2023-01-27T20:55:00","slug":"visualizing-geoip-information-in-kibana","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9745","title":{"rendered":"Visualizing GeoIP Information in Kibana"},"content":{"rendered":"

Before we can use map details in Kibana visualizations, we need to add fields with the geographic information. The first few steps are something the ELK admin staff will need to do in order to map source and\/or destination IPs to geographic information.<\/p>\n

First update the relevant index template to map the location information into geo-point fields \u2013 load this JSON (but, first, make sure there aren\u2019t existing<\/em> mappings otherwise you\u2019ll need to merge the existing JSON in with the new elements for geoip_src and geoip_dst<\/p>\n

{\r\n  \"_doc\": {\r\n    \"_meta\": {},\r\n    \"_source\": {},\r\n    \"properties\": {\r\n      \"geoip_dst\": {\r\n        \"dynamic\": true,\r\n        \"type\": \"object\",\r\n        \"properties\": {\r\n          \"ip\": {\r\n            \"type\": \"ip\"\r\n          },\r\n          \"latitude\": {\r\n            \"type\": \"half_float\"\r\n          },\r\n          \"location\": {\r\n            \"type\": \"geo_point\"\r\n          },\r\n          \"longitude\": {\r\n            \"type\": \"half_float\"\r\n          }\r\n        }\r\n      },\r\n      \"geoip_src\": {\r\n        \"dynamic\": true,\r\n        \"type\": \"object\",\r\n        \"properties\": {\r\n          \"ip\": {\r\n            \"type\": \"ip\"\r\n          },\r\n          \"latitude\": {\r\n            \"type\": \"half_float\"\r\n          },\r\n          \"location\": {\r\n            \"type\": \"geo_point\"\r\n          },\r\n          \"longitude\": {\r\n            \"type\": \"half_float\"\r\n          }\r\n        }\r\n      }\r\n    }\r\n  }\r\n}<\/pre>\n
First, click on the index template name to view the settings. Click to the \u2018mappings\u2019 tab and copy what is in there<\/p>\n
<\/p>\n
Munge in the two \u2018properties\u2019 in the above JSON. Edit the index template<\/p>\n
<\/p>\n
Click to the \u201cMappings\u201d section and use \u201cLoad JSON\u201d to import the new mapping configuration<\/p>\n
<\/p>\n
Paste in your JSON & click to \u201cLoad & Overwrite\u201d<\/p>\n
<\/p>\n
Voila \u2013 you will have geo-point items in the template.<\/p>\n
<\/p>\n
Next, the logstash pipeline needs to be configured to enrich log records with geoip information. There is a geoip filter available, which uses the MaxMind GeoIP database<\/a> (this is refreshed automatically<\/a>; currently, we do not merge in any geoip information for the private network address spaces) . You just need to indicate what field(s) have the IP address and where the location information should be stored. You can have multiple geographic IP fields \u2013 in this example, we map both source and destination IP addresses.<\/p>\n
        geoip {\r\n                source => \"src_ip\"\r\n                target => \"geoip_src\"\r\n                add_field => [ \"[geoip][location]\", \"%{[geoip][longitude]}\" ]\r\n                add_field => [ \"[geoip][location]\", \"%{[geoip][latitude]}\"  ]\r\n        }\r\n        geoip {\r\n                source => \"dst_ip\"\r\n                target => \"geoip_dest\"\r\n                add_field => [ \"[geoip][location]\", \"%{[geoip][longitude]}\" ]\r\n                add_field => [ \"[geoip][location]\", \"%{[geoip][latitude]}\"  ]\r\n        }<\/pre>\n
E.G.<\/p>\n
<\/p>\n
One logstash is restarted, the documents stored in Kibana will have geoip_src and geoip_dest fields:<\/p>\n
<\/p>\n
Once relevant data is being stored, use the refresh-looking button on the index pattern(s) to refresh the field list from stored data. This will add the geo-point items into the index pattern.<\/p>\n
<\/p>\n
Once GeoIP information is available in the index pattern, select the \u201cMaps\u201d visualization<\/p>\n
<\/p>\n
Leave the road map layer there (otherwise you won\u2019t see the countries!)<\/p>\n
<\/p>\n
Select \u2018Documents\u2019 as the data source to link in ElasticSearch data<\/p>\n
<\/p>\n
Select the index pattern that contains your data source (if your index pattern does not appear, then Kibana doesn\u2019t recognize the pattern as containing geographic fields \u2026 I\u2019ve had to delete and recreate my index pattern so the geographic fields were properly mapped).<\/p>\n
<\/p>\n
And select the field(s) that contain geographic details:<\/p>\n
<\/p>\n
You can name the layer<\/p>\n
<\/p>\n
And add a tool tip that will include the country code or name<\/p>\n
<\/p>\n
Under \u201cTerm joins\u201d, add a new join. Click on \u201cJoin –select–\u201d to link a field from the map to a field in your dataset.<\/p>\n
<\/p>\n
In this case, I am joining the two-character country codes —<\/p>\n
<\/p>\n
Normally, you can leave the \u201cand use metric count\u201d in place (the map is color coded by the number of requests coming from each country). If you want to add a filter, you can click the \u201cwhere — add filter –” link to edit the filter.<\/p>\n
<\/p>\n
In this example, I don\u2019t want to filter the data, so I\u2019ve left that at the default.<\/p>\n
<\/p>\n
Click \u201cSave & close\u201d to save the changes to the map visualization. To view your map, you won\u2019t find it under Visualizations \u2013 instead, click \u201cMaps\u201d along the left-hand navigation menu.<\/p>\n
<\/p>\n
Voila \u2013 a map where the shading on a country gets darker the more requests have come from the country.<\/p>\n
<\/p>\n
Internal Addresses<\/strong><\/p>\n
If we want to (and if we have information to map IP subnets to City\/State\/Zip\/LatLong, etc), we can<\/em> edit the database used for GeoIP mappings — https:\/\/github.com\/maxmind\/getting-started-with-mmdb<\/a> provides a perl module that interacts with the database file. That isn\u2019t currently done, so internal servers where traffic is sourced primarily from private address spaces won\u2019t have particularly thrilling map data.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
Before we can use map details in Kibana visualizations, we need to add fields with the geographic information. The first few steps are something the ELK admin staff will need to do in order to map source and\/or destination IPs to geographic information. First update the relevant index template to map the location information into …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1589,1800,1591,1643],"class_list":["post-9745","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elk","tag-geoip","tag-kibana","tag-logstash"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9745","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9745"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9745\/revisions"}],"predecessor-version":[{"id":9768,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9745\/revisions\/9768"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9745"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9745"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9745"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}