{"id":10799,"date":"2024-03-20T19:43:29","date_gmt":"2024-03-21T00:43:29","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=10799"},"modified":"2024-03-26T21:08:50","modified_gmt":"2024-03-27T02:08:50","slug":"opensearch-proof-of-concept-in-place-upgrade-from-elasticsearch-7-7-0-to-opensearch-2-12-0","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=10799","title":{"rendered":"OpenSearch Proof of Concept In-Place Upgrade from ElasticSearch 7.7.0 to OpenSearch 2.12.0"},"content":{"rendered":"\n

I need to migrate my ElasticSearch installation over to OpenSearch. From reading the documentation, it isn’t really clear if that is even possible as an in-place upgrade or if I’d need to use a remote reindex or snapshot backup\/restore. So I tested the process with a minimal data set. TL;DR: Yes, it works. <\/p>\n\n\n\n

Create a docker instance of ElasticSearch 7.7.0<\/h2>\n\n\n\n

mkdir \/docker\/es\/esdata
chmod -R g+dwx \/docker\/es\/esdata
chgrp -R 0 \/docker\/es\/esdata<\/p>\n\n\n\n

mkdir \/docker\/es\/esconfig<\/p>\n\n\n\n

Populate configuration info into .\/esconfig and .\/esdata is an empty directory<\/p>\n\n\n\n

docker run –name es770 -dit -v \/docker\/es\/esdata:\/usr\/share\/elasticsearch\/data<\/a> -v \/docker\/es\/esconfig:\/usr\/share\/elasticsearch\/config<\/a> -p 9200:9200 -p 9300:9300 -e “discovery.type=single-node” docker.elastic.co\/elasticsearch\/elasticsearch:7.7.0<\/a><\/p>\n\n\n\n

Populate Data into ElasticSearch Sandbox<\/h2>\n\n\n\n

Use curl to populate an index with some records \u2013 you can create lifecycle policies, customize the fields, etc … this is the bare minimum to validate that data in ES7.7 can be ingested by OS2.12curl -X POST “localhost:9200\/ljrtest\/_bulk” -H “Content-Type: application\/x-ndjson” -d’
{“index”: {“_id”: “1”}}
{“id”: “1”, “message”: “Record one”}
{“index”: {“_id”: “2”}}
{“id”: “2”, “message”: “Record two”}
{“index”: {“_id”: “3”}}
{“id”: “3”, “message”: “Record three”}
{“index”: {“_id”: “4”}}
{“id”: “4”, “message”: “Record four”}
{“index”: {“_id”: “5”}}
{“id”: “5”, “message”: “Record five”}
{“index”: {“_id”: “6”}}
{“id”: “6”, “message”: “Record six”}
{“index”: {“_id”: “7”}}
{“id”: “7”, “message”: “Record seven”}
{“index”: {“_id”: “8”}}
{“id”: “8”, “message”: “Record eight”}
{“index”: {“_id”: “9”}}
{“id”: “9”, “message”: “Record nine”}
{“index”: {“_id”: “10”}}
{“id”: “10”, “message”: “Record ten”}
‘<\/p>\n\n\n\n

Shut Down ElasticSearch<\/h2>\n\n\n\n

docker stop es770<\/p>\n\n\n\n

Bring Up an OpenSearch 2.12 Host<\/h2>\n\n\n\n

mkdir \/docker\/es\/osconfig<\/p>\n\n\n\n

Populate the configuration data for OpenSearch in .\/osconfig<\/p>\n\n\n\n

docker run –name os212 -dit -v \/docker\/es\/esdata:\/usr\/share\/opensearch\/data<\/a> -v \/docker\/es\/osconfig:\/usr\/share\/opensearch\/config -p 9200:9200 -p 9600:9600 -e “discovery.type=single-node” -e “OPENSEARCH_INITIAL_ADMIN_PASSWORD=P@s5w0rd-123” opensearchproject\/opensearch:2.12.0<\/p>\n\n\n\n

Verify Data is Still Available in OpenSearch<\/h2>\n\n\n\n

[root@docker es]# curl -k -u “admin:P@s5w0rd-123” https:\/\/localhost:9200\/ljrtest<\/a>
{“ljrtest”:{“aliases”:{},”mappings”:{“properties”:{“id”:{“type”:”text”,”fields”:{“keyword”:{“type”:”keyword”,”ignore_above”:256}}},”message”:{“type”:”text”,”fields”:{“keyword”:{“type”:”keyword”,”ignore_above”:256}}}}},”settings”:{“index”:{“creation_date”:”1710969477402″,”number_of_shards”:”1″,”number_of_replicas”:”1″,”uuid”:”AO5JBoyzSJiKZA9xeA2imQ”,”version”:{“created”:”7070099″,”upgraded”:”136337827″},”provided_name”:”ljrtest”}}}}<\/p>\n\n\n\n

Conclusion<\/h2>\n\n\n\n

Yes, a very basic data set in ElasticSearch 7.7.0 can be upgraded in-place to OpenSearch 2.12.0 — in the “real world” compatibility issues will crop up (flatten!!), but the idea is fundamentally sound. <\/p>\n\n\n\n

Problem, though, is<\/em> compatibility issues. We don’t have exotic data types in our instance but<\/em> Kibana uses “flatten” … so those rare people use use Kibana to access and visualize their data really cannot<\/em> just move to OpenSearch. That’s a huge caveat. I can recreate everything manually after deleting<\/em> all of the Kibana indices (and possibly some more, haven’t gone this route to see). But if I’m going to recreate everything, why wouldn’t I recreate everything and use remote reindex to move data? I can do this incrementally — take a week to move all the data slowly, do a catch-up reindex t-2 days, another t-1 days, another the day of the change, heck even one a few hours before the change. Then the change is a quick delta reindex, stop ElasticSearch, and swap over to OpenSearch. The backout is to just swing back to the fully functional, unchanged ElasticSearch instance. <\/p>\n","protected":false},"excerpt":{"rendered":"

I need to migrate my ElasticSearch installation over to OpenSearch. From reading the documentation, it isn’t really clear if that is even possible as an in-place upgrade or if I’d need to use a remote reindex or snapshot backup\/restore. So I tested the process with a minimal data set. TL;DR: Yes, it works. Create a …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,1589,1740],"class_list":["post-10799","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elk","tag-opensearch"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10799","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10799"}],"version-history":[{"count":2,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10799\/revisions"}],"predecessor-version":[{"id":10803,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10799\/revisions\/10803"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10799"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10799"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10799"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}