{"id":8925,"date":"2022-05-03T22:40:15","date_gmt":"2022-05-04T03:40:15","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=8925"},"modified":"2022-05-04T23:41:46","modified_gmt":"2022-05-05T04:41:46","slug":"elasticsearch-ilm-data-lifecycle","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=8925","title":{"rendered":"ElasticSearch ILM \u2013 Data Lifecycle"},"content":{"rendered":"<p>The following defines a simple data lifecycle policy we use for event log data.<\/p>\n<p>Immediately, the data is in the \u201chot\u201d phase.<\/p>\n<p>After one day, it is moved to the \u201cwarm\u201d phase where the number of segments is compressed to 1 (lots-o-segments are good for writing, but since we\u2019re dealing with timescale stats &amp; log data [i.e. something that\u2019s not being written to the next day], there is no need to optimize write performance. The index will be read only, thus can be optimized for read performance). After seven days, the index is frozen (mostly moved out of memory) as <em>in this use case<\/em>, data generally isn\u2019t used after a week. Thus, there is no need to fill up the server\u2019s memory to speed up access to unused data elements. Since freeze is deprecated in a future version (due to improvements in memory utilization that <em>should<\/em> obsolete freezing indices), we&#8217;ll need to watch our memory usage after upgrading to ES8.<\/p>\n<p>Finally, after fourteen days, the data is deleted.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"410\" height=\"504\" class=\"wp-image-8926\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image.png\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image.png 410w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-244x300.png 244w\" sizes=\"auto, (max-width: 410px) 100vw, 410px\" \/><\/p>\n<p>To use the policy, set it as the template on an index:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"427\" height=\"206\" class=\"wp-image-8927\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-1.png\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-1.png 427w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-1-300x145.png 300w\" sizes=\"auto, (max-width: 427px) 100vw, 427px\" \/><\/p>\n<p>Upon creating a new index (ljrlogs-5), the ILM policy has been applied:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"881\" height=\"508\" class=\"wp-image-8928\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-2.png\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-2.png 881w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-2-300x173.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-2-768x443.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/05\/word-image-2-750x432.png 750w\" sizes=\"auto, (max-width: 881px) 100vw, 881px\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The following defines a simple data lifecycle policy we use for event log data. Immediately, the data is in the \u201chot\u201d phase. After one day, it is moved to the \u201cwarm\u201d phase where the number of segments is compressed to 1 (lots-o-segments are good for writing, but since we\u2019re dealing with timescale stats &amp; log &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1624,1625],"class_list":["post-8925","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-ilm","tag-index-lifecycle-management"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8925","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8925"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8925\/revisions"}],"predecessor-version":[{"id":8929,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/8925\/revisions\/8929"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8925"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8925"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8925"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}