{"id":7864,"date":"2021-05-21T10:10:14","date_gmt":"2021-05-21T15:10:14","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=7864"},"modified":"2021-05-21T10:10:14","modified_gmt":"2021-05-21T15:10:14","slug":"confluent-kafka-queue-length","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=7864","title":{"rendered":"Confluent Kafka Queue Length"},"content":{"rendered":"<p>The <a href=\"https:\/\/docs.huihoo.com\/apache\/kafka\/confluent\/4.0\/clients\/confluent-kafka-python\/index.html#producer\" target=\"_blank\" rel=\"noopener\">documentation for the Python Confluent Kafka module<\/a> includes a len function on the producer. I wanted to use the function because we&#8217;re getting a number of duplicated messages on the client, and I was trying to isolate what might be causing the problem. Unfortunately, calling producer.len() failed indicating there&#8217;s no len() method. I used dir(producer) to show that, no, there <em>isn&#8217;t<\/em> a len() method.<\/p>\n<p>I realized today that the documentation is telling me that I can call the built-in len() function <em>on<\/em> a producer to get the queue length.<\/p>\n<p>Code:<\/p>\n<pre>print(f\"Before produce there are {len(producer)} messages awaiting delivery\")\r\nproducer.produce(topic, key=bytes(str(int(cs.timestamp) ), 'utf8'), value=cs.SerializeToString() )\r\nprint(f\"After produce there are {len(producer)} messages awaiting delivery\")\r\nproducer.poll(0) # Per https:\/\/github.com\/confluentinc\/confluent-kafka-python\/issues\/16 for queue full error\r\nprint(f\"After poll0 there are {len(producer)} messages awaiting delivery\")<\/pre>\n<p>Output:<\/p>\n<pre>Before produce there are 160 messages awaiting delivery\r\nAfter produce there are 161 messages awaiting delivery\r\nAfter poll0 there are 155 messages awaiting delivery<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>The documentation for the Python Confluent Kafka module includes a len function on the producer. I wanted to use the function because we&#8217;re getting a number of duplicated messages on the client, and I was trying to isolate what might be causing the problem. Unfortunately, calling producer.len() failed indicating there&#8217;s no len() method. I used &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33],"tags":[1362,1368,1361,664],"class_list":["post-7864","post","type-post","status-publish","format-standard","hentry","category-coding","tag-apache-kafka","tag-confluent-kafka","tag-kafka","tag-python"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7864","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7864"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7864\/revisions"}],"predecessor-version":[{"id":7865,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7864\/revisions\/7865"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7864"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7864"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7864"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}