{"id":11069,"date":"2024-07-22T19:12:00","date_gmt":"2024-07-23T00:12:00","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=11069"},"modified":"2024-07-26T11:18:25","modified_gmt":"2024-07-26T16:18:25","slug":"migrating-redis-data","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=11069","title":{"rendered":"Migrating Redis Data"},"content":{"rendered":"\n

So, I know that Redis should be a data cache that can be repopulated … but we use it to calculate deltas (what was the value last<\/em> time) … so repopulating the information makes the first half hour or so of calculations rather slow as the application tries redis, gets nothing, and fails back to a database query. Then we get a backlog of data to churn through, and it would just be better<\/em> if the Redis cache hadn’t gone away in the first place. And if you own both servers and the files are in the same format, you could just copy the cache db from the old server to the new one. But … when you cannot just copy the file and<\/em> you would really prefer the data not disappear and need to be repopulated … there’s a script for that! This python script reads all of the data from the “old” server and populates it into the “new” server. <\/p>\n\n\n

\nimport redis\n\ndef migrate_data(redis_source_host, redis_source_port, redis_source_db, redis_source_password,\n                 redis_dest_host, redis_dest_port, redis_dest_db, redis_dest_password):\n    # Connect to the source Redis server\n    source_client = redis.StrictRedis(host=redis_source_host, port=redis_source_port, db=redis_source_db, password=redis_source_password)\n\n    # Connect to the destination Redis server\n    dest_client = redis.StrictRedis(host=redis_dest_host, port=redis_dest_port, db=redis_dest_db, password=redis_dest_password)\n\n    # Fetch all keys from the source Redis\n    keys = source_client.keys('*')\n\n    for key in keys:\n        # Get the type of the key\n        key_type = source_client.type(key).decode('utf-8')\n\n        if key_type == 'string':\n            value = source_client.get(key)\n            print("Setting string value in dest")\n            dest_client.set(key, value)\n        elif key_type == 'list':\n            values = source_client.lrange(key, 0, -1)\n            print("Setting list value in dest")\n            dest_client.delete(key)  # Ensure the list is empty before pushing\n            for value in values:\n                dest_client.rpush(key, value)\n        elif key_type == 'set':\n            values = source_client.smembers(key)\n            print("Setting set value in dest")\n            dest_client.delete(key)  # Ensure the set is empty before pushing\n            for value in values:\n                dest_client.sadd(key, value)\n        elif key_type == 'zset':\n            values = source_client.zrange(key, 0, -1, withscores=True)\n            print("Setting zset value in dest")\n            dest_client.delete(key)  # Ensure the zset is empty before pushing\n            for value, score in values:\n                dest_client.zadd(key, {value: score})\n        elif key_type == 'hash':\n            values = source_client.hgetall(key)\n            print("Setting hash value in dest")\n            dest_client.delete(key)  # Ensure the hash is empty before pushing\n            dest_client.hmset(key, values)\n\n    print("Data migration completed.")\n\nif __name__ == "__main__":\n    # Source Redis server details\n    redis_source_host = 'oldredis.example.com'\n    redis_source_port = 6379\n    redis_source_db = 0\n    redis_source_password = 'SourceRedisPassword'\n\n    # Destination Redis server details\n    redis_dest_host = 'newredis.example.com'\n    redis_dest_port = 6379\n    redis_dest_db = 0\n    redis_dest_password = 'DestRedisPassword'\n\n    # Migrate data\n    migrate_data(redis_source_host, redis_source_port, redis_source_db, redis_source_password,\n                 redis_dest_host, redis_dest_port, redis_dest_db, redis_dest_password)\n\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"
So, I know that Redis should be a data cache that can be repopulated … but we use it to calculate deltas (what was the value last time) … so repopulating the information makes the first half hour or so of calculations rather slow as the application tries redis, gets nothing, and fails back to …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1945,30],"tags":[2032,664,427],"class_list":["post-11069","post","type-post","status-publish","format-standard","hentry","category-python","category-system-administration","tag-data-migration","tag-python","tag-redis"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11069","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11069"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11069\/revisions"}],"predecessor-version":[{"id":11070,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11069\/revisions\/11070"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11069"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11069"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11069"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}