{"id":7869,"date":"2021-05-21T13:31:47","date_gmt":"2021-05-21T18:31:47","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=7869"},"modified":"2021-05-21T13:31:47","modified_gmt":"2021-05-21T18:31:47","slug":"kafka-troubleshooting-for-those-who-enjoy-reading-network-traces","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=7869","title":{"rendered":"Kafka Troubleshooting (for those who enjoy reading network traces)"},"content":{"rendered":"<p>I finally had a revelation that allowed me to definitively prove that I am <em>not<\/em> doing anything strange that is causing duplicated messages to appear in the Kafka stream &#8212; it&#8217;s a clear text protocol! That means you can use Wireshark, tcpdump, etc to capture everything that goes over the wire. This shows that the GUID I generated for the duplicated message <em>only<\/em> appears one time in the network trace. Whatever funky stuff is going on that makes the client see it twice? Not me <span class=\"emoji\">\ud83d\ude0a<\/span><\/p>\n<p>I used tcpdump because the batch server doesn&#8217;t have tshark (and it&#8217;s not <em>my<\/em> server, so I&#8217;m not going to go requesting additional binaries if there&#8217;s something sufficient for my need already available). Ran <strong>tcpdump -w \/srv\/data\/ljr.cap port 9092<\/strong> to grab everything that transits port 9092 while my script executed. Once the batch completed, I stopped tcpdump and transferred the file over to my workstation to view the capture in Wireshark. Searched the packet bytes for my duplicated GUID &#8230; and there&#8217;s only one.<\/p>\n<p><a href=\"https:\/\/www.rushworth.us\/lisa\/?attachment_id=7871\" rel=\"attachment wp-att-7871\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-7871\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-1024x355.png\" alt=\"\" width=\"960\" height=\"333\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-1024x355.png 1024w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-300x104.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-768x266.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-1536x532.png 1536w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace-750x260.png 750w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2021\/05\/KafkaGuidOnlyAppearsOnceInNetworkTrace.png 1910w\" sizes=\"auto, (max-width: 960px) 100vw, 960px\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I finally had a revelation that allowed me to definitively prove that I am not doing anything strange that is causing duplicated messages to appear in the Kafka stream &#8212; it&#8217;s a clear text protocol! That means you can use Wireshark, tcpdump, etc to capture everything that goes over the wire. This shows that the &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33],"tags":[1362,1361],"class_list":["post-7869","post","type-post","status-publish","format-standard","hentry","category-coding","tag-apache-kafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7869","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7869"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7869\/revisions"}],"predecessor-version":[{"id":7872,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/7869\/revisions\/7872"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7869"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7869"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7869"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}