{"id":9502,"date":"2022-09-07T21:31:53","date_gmt":"2022-09-08T02:31:53","guid":{"rendered":"https:\/\/www.rushworth.us\/lisa\/?p=9502"},"modified":"2022-10-07T15:32:45","modified_gmt":"2022-10-07T20:32:45","slug":"upgrading-kafka-from-2-5-0-to-3-2-3","status":"publish","type":"post","link":"https:\/\/www.rushworth.us\/lisa\/?p=9502","title":{"rendered":"Upgrading Kafka from 2.5.0 to 3.2.3"},"content":{"rendered":"

Bidirectional backwards compatibility was introduced in 2017<\/a> \u2013 which means my experience where you needed to upgrade the broker first<\/em> and then the clients is no longer true. Rejoice!<\/p>\n

<\/a>Sandbox Setup<\/h4>\n

Two CentOS docker containers were provisioned as follows:<\/p>\n

docker run -dit --name=kafka1 -p 9092:9092 centos:latest\r\ndocker run -dit --name=kafka2 -p 9093:9092 -p9000:9000 centos:latest<\/pre>\n
# Shell into each container and do the following:<\/p>\n
sed -i -e \"s|mirrorlist=|#mirrorlist=|g\" \/etc\/yum.repos.d\/CentOS-*\r\nsed -i -e \"s|#baseurl=http:\/\/mirror.centos.org|baseurl=http:\/\/vault.centos.org|g\" \/etc\/yum.repos.d\/CentOS-*<\/pre>\n
# Get Ips and hosts into \/etc\/hosts<\/p>\n
172.17.0.2 40c2222cfea0
\n172.17.0.3 2923addbcb6d<\/p>\n
# Update installed packages & install required tools<\/p>\n
dnf update\r\nyum install -y passwd vim net-tools wget git unzip<\/pre>\n
# Add a kafka user, make a kafka folder, and give the kafka user ownership of the kafka folder<\/pre>\n
useradd kafka\r\npasswd kafka\r\nusermod -aG wheel kafka\r\n\r\nmkdir \/kafka\r\n\r\nchown kafka:kafka \/kafka<\/pre>\n
# Install Kafka<\/p>\n
su \u2013 kafka\r\ncd \/kafka\r\nwget https:\/\/archive.apache.org\/dist\/kafka\/2.5.0\/kafka_2.12-2.5.0.tgz<\/a>\r\ntar vxzf kafka_2.12-2.5.0.tgz\r\nrm kafka_2.12-2.5.0.tgz\r\nln -s \/kafka\/kafka_2.12-2.5.0 \/kafka\/kafka<\/pre>\n
# Configure zookeeper<\/p>\n
vi \/kafka\/kafka\/config\/zookeeper.properties\r\ndataDir=\/kafka\/zookeeperdata\r\nserver.1=172.17.0.2:2888:3888<\/pre>\n
# Start Zookeeper on the first server<\/p>\n
screen -S zookeeper\r\n\/kafka\/kafka\/bin\/zookeeper-server-start.sh \/kafka\/kafka\/config\/zookeeper.properties<\/pre>\n
# Configure the cluster<\/p>\n
vi \/kafka\/kafka\/config\/server.properties\r\n\r\nbroker.id=1 # unique number per cluster node\r\nlisteners=PLAINTEXT:\/\/:9092\r\nzookeeper.connect=172.17.0.2:2181<\/pre>\n
# Start Kafka<\/p>\n
screen -S kafka\r\n\/kafka\/kafka\/bin\/kafka-server-start.sh \/kafka\/kafka\/config\/server.properties<\/pre>\n
# Edit producer.properties on a server<\/p>\n
vi \/kafka\/kafka\/config\/producer.properties\r\nbootstrap.servers=172.17.0.2:9092,172.17.0.3:9092<\/pre>\n
# Create test topic<\/p>\n
\/kafka\/kafka\/bin\/kafka-topics.sh --create --zookeeper 172.17.0.2:2181 --replication-factor 2 --partitions 1 --topic ljrTest<\/pre>\n
# Post messages to the topic<\/p>\n
\/kafka\/kafka\/bin\/kafka-console-producer.sh --broker-list 172.17.0.2:9092 --producer.config \/kafka\/kafka\/config\/producer.properties --topic ljrTest<\/pre>\n
# Retrieve messages from topic<\/p>\n
\/kafka\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 172.17.0.2:9092 --topic ljrTest --from-beginning\r\n\/kafka\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 172.17.0.3:9092 --topic ljrTest --from-beginning<\/pre>\n
Voila, a functional Kafka sandbox cluster.<\/p>\n
Now we\u2019ll install the cluster manager<\/p>\n
cd \/kafka\r\ngit clone --depth 1 --branch 3.0.0.6 https:\/\/github.com\/yahoo\/CMAK.git\r\ncd CMAK\r\nvi conf\/application.conf\r\ncmak.zkhosts=\"40c2222cfea0:2181\"\r\n\r\n# CMAK requires java > 1.8 \u2026 so getting 11 set up\r\ncd \/usr\/lib\/jvm\r\nwget https:\/\/cdn.azul.com\/zulu\/bin\/zulu11.58.23-ca-jdk11.0.16.1-linux_x64.zip\r\nunzip zulu11.58.23-ca-jdk11.0.16.1-linux_x64.zip\r\nmv zulu11.58.23-ca-jdk11.0.16.1-linux_x64 zulu-11\r\nPATH=\/usr\/lib\/jvm\/zulu-11\/bin:$PATH\r\n\r\n.\/sbt -java-home \/usr\/lib\/jvm\/zulu-11 clean dist\r\n\r\ncp \/kafka\/CMAK\/target\/universal\/cmak-3.0.0.6.zip \/kafka\r\n\r\ncd \/kafka\r\nunzip cmak-3.0.0.6.zip\r\ncd cmak-3.0.0.6\r\nscreen -S CMAK\r\nbin\/cmak -java-home \/usr\/lib\/jvm\/zulu-11 -Dconfig.file=\/kafka\/cmak-3.0.0.6\/conf\/application.conf -Dhttp.port=9000<\/pre>\n
Access it at http:\/\/cmak_host:9000<\/a><\/p>\n
<\/a>Sandbox Upgrade Process<\/h4>\n
# Back up the Kafka installation (excluding log files)<\/p>\n
tar cvfzp \/kafka\/kafka-2.5.0.tar.gz --exclude logs \/kafka\/ws_npm_kafka\/kafka_2.12-2.5.0<\/pre>\n
# Get newest Kafka version installed
\n# From another host where you can download the file, transfer it to the kafka server<\/p>\n
scp kafka_2.12-3.2.3.tgz list@kafka1:\/tmp\/<\/pre>\n
# Back on the Kafka server — copy the tgz file into the Kafka directory<\/p>\n
mv \/tmp\/kafka_2.12-3.2.3.tgz \/kafka\/kafka<\/pre>\n
# Verify Kafka data is stored outside of the install directory:<\/p>\n
[kafka@40c2222cfea0 config]$ grep log.dir server.properties\r\nlog.dirs=\/tmp\/kafka-logs<\/pre>\n
# Verify zookeeper data is stored outside of the install directory:<\/p>\n
[kafka@40c2222cfea0 config]$ grep dataDir zookeeper.properties\r\ndataDir=\/kafka\/zookeeperdata<\/pre>\n
# Get the new version of Kafka \u2013 start with the zookeeper(s) then do the other nodes<\/p>\n
cd \/kafka\r\nwget https:\/\/downloads.apache.org\/kafka\/3.2.3\/kafka_2.12-3.2.3.tgz\r\ntar vxfz \/kafka\/kafka_2.12-3.2.3.tgz<\/pre>\n
# Copy config from old iteration to new<\/p>\n
cp \/kafka\/kafka_2.12-2.5.0\/config\/* \/kafka\/kafka_2.12-3.2.3\/config\/<\/pre>\n
# Edit server.properties and add a configuration line to force the inter-broker protocol version to the currently running<\/em> Kafka version
\n# This ensures your cluster is using the \u201cold\u201d version to communicate and you can, if needed, revert to the previous version<\/p>\n
vi \/kafka\/kafka\/config\/server.properties\r\ninter.broker.protocol.version=2.5.0<\/pre>\n
# Restart each Kafka server \u2013 waiting until it has come online before restarting the next one \u2013 with the new binaries
\n# Stop kafka<\/p>\n
systemctl stop kafka<\/pre>\n
# Move symlink to new folder<\/p>\n
unlink \/kafka\/kafka\r\nln -s \/kafka\/kafka_2.12-3.2.3 \/kafka\/kafka<\/pre>\n
# start kafka<\/p>\n
systemctl start kafka<\/pre>\n
# Or, to watch it run,<\/p>\n
\/kafka\/kafka\/bin\/kafka-server-start.sh \/kafka\/kafka\/config\/server.properties<\/pre>\n
# Finally, ensure you\u2019ve still got \u2018stuff\u2019<\/p>\n
\/kafka\/kafka\/bin\/kafka-console-consumer.sh --bootstrap-server 172.17.0.3:9092 --topic ljrTest --from-beginning<\/pre>\n
# And verify the version has updated<\/p>\n
[kafka@40c2222cfea0 bin]$ .\/kafka-topics.sh --version\r\n3.2.3 (Commit:50029d3ed8ba576f)<\/pre>\n
# Until this point, we can just roll back to the old folder & revert to the previous version of Kafka \u2026 that\u2019s out backout plan.<\/p>\n
# Once everything has been confirmed to be working, bump the inter-broker protocol version to the new version & restart Kafka<\/p>\n
vi \/kafka\/kafka\/config\/server.properties\r\ninter.broker.protocol.version=3.2<\/pre>\n","protected":false},"excerpt":{"rendered":"
Bidirectional backwards compatibility was introduced in 2017 \u2013 which means my experience where you needed to upgrade the broker first and then the clients is no longer true. Rejoice! Sandbox Setup Two CentOS docker containers were provisioned as follows: docker run -dit –name=kafka1 -p 9092:9092 centos:latest docker run -dit –name=kafka2 -p 9093:9092 -p9000:9000 centos:latest # …<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[1361],"class_list":["post-9502","post","type-post","status-publish","format-standard","hentry","category-system-administration","tag-kafka"],"_links":{"self":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9502","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9502"}],"version-history":[{"count":1,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9502\/revisions"}],"predecessor-version":[{"id":9503,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=\/wp\/v2\/posts\/9502\/revisions\/9503"}],"wp:attachment":[{"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9502"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9502"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rushworth.us\/lisa\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9502"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}