{"id":9566,"date":"2022-10-28T00:04:53","date_gmt":"2022-10-28T05:04:53","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9566"},"modified":"2022-10-28T00:04:53","modified_gmt":"2022-10-28T05:04:53","slug":"unable-to-use-jmx-remotely-for-kafka-stats","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9566","title":{"rendered":"Unable to Use JMX Remotely for Kafka Stats"},"content":{"rendered":"\n<p>I noticed, today, that our Kafka Manager interface only shows details from one server &#8212; the one where we run Kafka Manager. We&#8217;ve done everything that we need to do in order to get this working &#8212; the port shows as open with nmap, the command to run Kafka includes all of the settings. I&#8217;ve even tried setting the JMX hostname, but still there is just one server reporting data<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts.png\"><img loading=\"lazy\" decoding=\"async\" width=\"834\" height=\"525\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts.png\" alt=\"\" class=\"wp-image-9567\" srcset=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts.png 834w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts-300x189.png 300w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts-768x483.png 768w, https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NoJMXForRemoteHosts-750x472.png 750w\" sizes=\"auto, (max-width: 834px) 100vw, 834px\" \/><\/a><\/figure>\n\n\n\n<p>Then I happened across an <a rel=\"noreferrer noopener\" href=\"https:\/\/olegz.wordpress.com\/2009\/03\/23\/jmx-connectivity-through-the-firewall\/\" target=\"_blank\">article online that detailed how JMX actually uses <em>three<\/em> ports<\/a> &#8212; the configured port 9999 and two other randomly selected and non-configurable ports. I used netstat to list <em>all<\/em> of the ports in use by the Java PID running my Kafka server and, voila, there were two odd-ball high ports (30000&#8217;s and 40000&#8217;s). I added those additional ports to the firewall rules and &#8230; I&#8217;ve got data for all of the Kafka servers!<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NowThereIsData.png\"><img loading=\"lazy\" decoding=\"async\" width=\"229\" height=\"261\" src=\"https:\/\/www.rushworth.us\/lisa\/wp-content\/uploads\/2022\/10\/KafkaManager-NowThereIsData.png\" alt=\"\" class=\"wp-image-9568\"\/><\/a><\/figure>\n\n\n\n<p>This is obviously a short-term solution as the two randomly selected ports will be <em>different<\/em> when I restart the service next time. I&#8217;d prefer to leave the firewall in place (i.e. <em>not<\/em> just open all ports >1024 between the Kafka Manager host and all of the Kafka servers) so might put together a script to identify the &#8220;oddball&#8221; ports associated to the Java pid and add them to transient firewalld rules. But the last server restart was back in 2021 &#8230; so I might just manually add them after the upgrade next week and worry about something &#8216;better&#8217; next year!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I noticed, today, that our Kafka Manager interface only shows details from one server &#8212; the one where we run Kafka Manager. We&#8217;ve done everything that we need to do in order to get this working &#8212; the port shows as open with nmap, the command to run Kafka includes all of the settings. I&#8217;ve &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[1737,1760,1361,294],"class_list":["post-9566","post","type-post","status-publish","format-standard","hentry","category-system-administration","tag-firewall","tag-jmx","tag-kafka","tag-linux"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9566","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9566"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9566\/revisions"}],"predecessor-version":[{"id":9569,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9566\/revisions\/9569"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9566"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9566"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9566"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}