Get rid of Zookeeper and use Kraft

It this post, i will explain how to be able to use Kraft and why use it.

Kraf is a concensus protocol allowing Kafka to work independently without Zookeeper.

This simplifies the fact there is nore more need of two different services (Kafka and Zookeeper) to share metadata. It also enables kafka fail over control to be almost instant. Thus the effect of kafka start and stop is faster.

The feature began with the version Kafka 2.8 and is enhanced with every new version BUT for the moment (June 2022), this solution is not yet production ready.

Of course this changes a little the infrastructure and the connection method.

	With Zookeeper	Without Zookeeper
Client and service configuration	zookeeper.connect=zookeeper:2181	bootstrap.servers=broker:9092
Schema registry configuration	kafkastore.connection.url=zookeeper:2181	kafkastore.bootstrap.servers=broker:9092
Kafka admin tool	kafka-topics –zookeeper zookeeper:2181	kafka-topics –bootstrap-server broker:9092 … –command-config properties to connect to brokers
REST Proxy API	v1	v2 and v3
Get Cluster ID	zookeeper-shell zookeeper:2181 get/cluster/id	kafka-metadata-quorum or view metadata.properties or confluent cluster describe --url http://broker:8090 --output json

How to configuration and start Kafka with Kraft

Generate a UID:

./bin/kafka-storage.sh random-uuid
xtzWWN4bTjitpL3kfd9s5g

Format the data directory to be compatible with Kraft (to run on each Kafka broker), do not forget to set the cluster ID.

./bin/kafka-storage.sh format -t <uuid> -c ./config/kraft/server.properties
Formatting /tmp/kraft-combined-logs --cluster-id XXXXXXXXXX

In the file server.properties, do not forget to set all ther brokers hosts in the line “controller.quorum.voters”

process.roles=controller
node.id=1
listeners=CONTROLLER://controller1.example.com:9093
controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093

Start Kafka with pointing to kraft config file

./bin/kafka-server-start.sh ./config/kraft/server.properties

You are now ready to work with a broker running with no zookeeper dependencies to be healty.