{"id":10851,"date":"2024-04-01T21:16:53","date_gmt":"2024-04-02T02:16:53","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=10851"},"modified":"2024-04-03T21:23:18","modified_gmt":"2024-04-04T02:23:18","slug":"elasticsearch-to-opensearch-migration-remote-reindex-to-move-data","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=10851","title":{"rendered":"ElasticSearch to OpenSearch Migration: Remote Reindex to Move Data"},"content":{"rendered":"\n

Since we cannot do an in-place upgrade of our ElasticSearch environment, I need to move<\/em> everything to the new servers. The biggest component is moving the data — which can easily be done using the remote reindex. Use the ElasticSearch API to get a list of all indices, and tell the OpenSearch API to reindex that index from the ElasticSearch remote. This operates on deltas — it will add new documents to an index — so my plan is to spend a few days seeding the initial data, then perform delta updates leading up to the scheduled change. <\/p>\n\n\n

\nimport requests\nfrom requests.auth import HTTPBasicAuth\n\nf = open("results.txt", "a")\n\nlistIndexNames = []\n\nreqGetIndexes = requests.get('https:\/\/elasticsearch.example.com:9200\/_cat\/indices?format=json', auth=HTTPBasicAuth('something','something'), verify=False)\nfor jsonIndex in reqGetIndexes.json():\n        if jsonIndex.get('index')[0] != '.':\n                listIndexNames.append(jsonIndex.get('index'))\n\nfor strIndexName in listIndexNames:\n  jsonReindexItem = {\n    "source": {\n      "remote": {\n        "host": "https:\/\/elasticsearch.example.com:9200",\n        "username": "something",\n        "password": "something"\n      },\n  "index": strIndexName\n    },\n    "dest": {\n  "index": strIndexName\n    }\n  }\n\n  r = requests.post('https:\/\/opensearch.example.com:9200\/_reindex', json=jsonReindexItem, auth = HTTPBasicAuth('something', 'something'), verify=False)\n  print(r.json())\n  jsonResponse = r.json()\n\n  if r.status_code == 400 and "mapping set to strict" in jsonResponse.get('failures')[0].get('cause').get("reason"):\n    # {'error': {'root_cause': [{'type': 'x_content_parse_exception', 'reason': '[1:2] [reindex] unknown field [key]'}], 'type': 'x_content_parse_exception', 'reason': '[1:2] [reindex] unknown field [key]'}, 'status': 400}\n    if jsonResponse.get('failures'):\n      print(jsonResponse.get('failures')[0].get('cause').get("reason"))\n      print("I need to set dynamic mapping")\n      r2 = requests.put(f'https:\/\/opensearch.example.com:9200\/{strIndexName}\/_mapping', json={"dynamic":"true"}, auth = HTTPBasicAuth('something', 'something'), verify=False)\n      print(r2.json)\n      r3 = requests.post('https:\/\/opensearch.example.com:9200\/_reindex', json=jsonReindexItem, auth = HTTPBasicAuth('something', 'something), verify=False)\n      print(r.json())\n      print(f"{strIndexName}\\t{r3.status_code}\\t{r.json()}\\n")\n      f.write(f"{strIndexName}\\t{r3.status_code}\\t{r.json()}\\n")\n\n  elif r.status_code == 200:\n    print(jsonResponse)\n    print(f"{strIndexName}\\t{r.status_code}\\t{r.json()}\\n")\n    f.write(f"{strIndexName}\\t{r.status_code}\\t{r.json()}\\n")\n  else:\n    print(f"HTTP Error: {r.status_code} on web call")\n    print(f"{strIndexName}\\t{r.status_code}\\t{r.json()}\\n")\n    f.write(f"{strIndexName}\\t{r.status_code}\\t{r.json()}\\n")\n\nf.close()\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"
Since we cannot do an in-place upgrade of our ElasticSearch environment, I need to move everything to the new servers. The biggest component is moving the data — which can easily be done using the remote reindex. Use the ElasticSearch API to get a list of all indices, and tell the OpenSearch API to reindex …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1588],"tags":[1590,2001,1740],"class_list":["post-10851","post","type-post","status-publish","format-standard","hentry","category-elk","tag-elasticsearch","tag-elasticsearch-to-opensearch-migration","tag-opensearch"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10851","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10851"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10851\/revisions"}],"predecessor-version":[{"id":10852,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/10851\/revisions\/10852"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10851"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10851"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10851"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}