{"id":11179,"date":"2024-09-23T10:25:05","date_gmt":"2024-09-23T15:25:05","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=11179"},"modified":"2024-09-25T10:40:51","modified_gmt":"2024-09-25T15:40:51","slug":"kafka-streams-consumer-groups-and-stickiness","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=11179","title":{"rendered":"Kafka Streams, Consumer Groups, and Stickiness"},"content":{"rendered":"\n<p>The Java application I recently inherited had a lot of &#8230; quirks. One of the strangest was that it calculated throughput statistics based on &#8216;start&#8217; values in a cache that was only refreshed every four hours. So at a minute past the data refresh, the throughput is averaged out over that minute. At three hours and fifty nine minutes past the data refresh, the throughput is averaged out over three hours and fifty nine minutes. In the process of correcting <em>this<\/em> (reading directly from the cached data rather than using an in-memory copy of the cached data), I noticed that the running application paused <em>a lot<\/em> as the Kafka group was re-balanced. <\/p>\n\n\n\n<p>Which is especially odd because I&#8217;ve got a stable number of clients in each consumer group. But pods restart occasionally, and there was nothing done to attempt to stabilize partition assignment. <\/p>\n\n\n\n<p>Which was odd because Kafka has had mechanisms to reduce re-balancing &#8212; StickyAssignor added in 0.11<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; title: ; notranslate\" title=\"\">\n        \/\/ Set the partition assignment strategy to StickyAssignor\n        config.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, &quot;org.apache.kafka.clients.consumer.StickyAssignor&quot;);\n\n<\/pre><\/div>\n\n\n<p>And groupInstanceId in 2.3.0<\/p>\n\n\n<div class=\"wp-block-syntaxhighlighter-code \"><pre class=\"brush: java; title: ; notranslate\" title=\"\">\n        \/\/ Set the group instance ID\n        String groupInstanceId = UUID.randomUUID().toString();\n        config.put(ConsumerConfig.GROUP_INSTANCE_ID_CONFIG, groupInstanceId);\n<\/pre><\/div>\n\n\n<p>Now, I&#8217;m certain that a UUID isn&#8217;t the <em>best<\/em> way to go about crafting your group instance ID name &#8230; but it produces a &#8220;name&#8221; that isn&#8217;t likely to be duplicated. Since deploying this change, I went from seeing three or four re-balance operations an hour to <em>zero<\/em>. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>The Java application I recently inherited had a lot of &#8230; quirks. One of the strangest was that it calculated throughput statistics based on &#8216;start&#8217; values in a cache that was only refreshed every four hours. So at a minute past the data refresh, the throughput is averaged out over that minute. At three hours &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1547,1762],"tags":[],"class_list":["post-11179","post","type-post","status-publish","format-standard","hentry","category-java","category-kafka"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11179"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11179\/revisions"}],"predecessor-version":[{"id":11180,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/11179\/revisions\/11180"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}