{"id":11187,"date":"2024-10-04T11:20:20","date_gmt":"2024-10-04T16:20:20","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=11187"},"modified":"2024-10-04T11:20:21","modified_gmt":"2024-10-04T16:20:21","slug":"watching-redis-records-for-strange-changes","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=11187","title":{"rendered":"Watching Redis Records for Strange Changes"},"content":{"rendered":"\n

I write a lot of things down to save myself time the next<\/em> time I need to do the same sort of thing — and publish this to the Internet in case I can save someone else<\/em> time too. But this one is so specific, I’m not sure it’s an “ever going to encounter this again” sort of thing. Just in case, though — I have device data being stored in redis — because the device doesn’t know its throughput values, you need the last<\/em> time and last<\/em> value paired with the current device metrics to calculate throughput. OK. But, sporadically, the cached data is updated insomuch as a new record is posted with a new timestamp. But the actual values, other than timestamp, remain unchanged. With millions of interfaces, it’s challenging to identify these situations by spot-checking the visualizations. Instead, I need to monitor redis and identify when the tstamp is updated but no other values change. <\/p>\n\n\n

\nimport redis\nimport time\nimport re\nimport json\nimport os\n\n# Configuration\nredis_host = 'redishost.example.net'\nredis_port = 6379\nredis_password = 'P@5sw0rDG03sH3r3'  # Replace with your Redis password\npattern = re.compile(r'INTERFACE_RAW_STATS_hostname\\d\\d\\d\\d_\\d+_\\d+')\noutput_file = 'changed_records.json'\n\n# Connect to Redis\nclient = redis.StrictRedis(host=redis_host, port=redis_port, password=redis_password, decode_responses=True)\n\n# Dictionary to track records\nrecords = {}\nmatching_keys = []\n\ndef get_matching_keys():\n    """\n    Retrieve keys from Redis matching the specified pattern.\n\n    Returns:\n        list: A list of keys that match the pattern.\n    """\n    all_keys = client.keys()\n    matching_keys = [key for key in all_keys if pattern.match(key)]\n    return matching_keys\n\ndef process_keys():\n    """\n    Process Redis keys to track changes in data.\n\n    Retrieves keys matching the pattern, gets their data using HGETALL,\n    and tracks changes. If only the 'tstamp' field has changed and all\n    other fields remain the same, the record is written to a file.\n    """\n    global records\n    i = 0\n\n    for key in matching_keys:\n        i += 1\n        data = client.hgetall(key)\n        if i == 1 or i % 1000 == 0:\n            print(f"Processed {i} records")\n\n        if not data:\n            continue\n\n        collector_name = data.get('collectorName')\n        node_id = data.get('nodeId')\n        if_index = data.get('ifIndex')\n        tstamp = data.get('tstamp')\n\n        if not collector_name or not node_id or not if_index or not tstamp:\n            continue\n\n        unique_key = f"{collector_name}_{node_id}_{if_index}"\n\n        if unique_key in records:\n            previous_data = records[unique_key]\n            if previous_data['tstamp'] != tstamp:\n                # Check if all other values are the same\n                if all(data[k] == previous_data[k] for k in data if k != 'tstamp'):\n                    print(f"***** Record changed: {json.dumps(data, indent=2)} *****")\n                    write_to_file(data)\n            records[unique_key] = data  # Update the record\n        else:\n            records[unique_key] = data\n\ndef write_to_file(data):\n    """\n    Write the given data to a file.\n\n    Args:\n        data (dict): The data to write to the file.\n    """\n    with open(output_file, 'a') as file:\n        file.write(json.dumps(data) + '\\n')\n\nif __name__ == "__main__":\n    # Ensure the output file is empty at the start\n    if os.path.exists(output_file):\n        os.remove(output_file)\n\n    # Retrieve the list of matching keys once\n    matching_keys = get_matching_keys()\n\n    while True:\n        process_keys()\n        print("Sleeping ... ")\n        time.sleep(300)  # Sleep for 5 minutes\n\n<\/pre><\/div>","protected":false},"excerpt":{"rendered":"
I write a lot of things down to save myself time the next time I need to do the same sort of thing — and publish this to the Internet in case I can save someone else time too. But this one is so specific, I’m not sure it’s an “ever going to encounter this …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1945],"tags":[664,427],"class_list":["post-11187","post","type-post","status-publish","format-standard","hentry","category-python","tag-python","tag-redis"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11187","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11187"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11187\/revisions"}],"predecessor-version":[{"id":11188,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11187\/revisions\/11188"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11187"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11187"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}