This is a brief guide on a Multi Node Kafka Cluster Setup using Windows Docker:
Setting Up a Three Node Kafka Cluster
1- Start ZooKeeper and Kafka using Docker Compose up command
2- In another terminal window, go to the same directory (kafka-cluster). Before we move on, let’s make sure the services are up and running
3- Check the ZooKeeper logs to verify that ZooKeeper is healthy. For example, for service zookeeper-1
docker logs <zookeeper-1_containerId>
** repeat this step to varify the rest of the zookeeper containers
4- Verify that ZK ensemble is ready
docker run –net=host –rm confluentinc/cp-zookeeper:3.3.1 bash -c “echo stat | nc localhost <ZOOKEEPER_CLIENT_PORT> | grep Mode”
** repeat this step to varify the rest of the Zookeeper containers
Output: You should see one leader and two follower:
5- Check the logs to see the broker has booted up successfully.
docker logs <kafka-1_containerId>
docker logs <kafka-2_containerId>
docker logs <kafka-3_containerId>
6- Test that the broker is working as expected.
Now that the brokers are up, we’ll test that they’re working as expected by creating a topic.
docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –create –topic bar –partitions 3 –replication-factor 3 –if-not-exists –zookeeper localhost:32181
You should see the following output:
Created topic “bar”
7- Now verify that the topic is created successfully by describing the topic.
docker run –net=host –rm confluentinc/cp-kafka:latest kafka-topics –describe –topic bar –zookeeper localhost:32181
8- Next, we’ll try generating some data to the bar topic we just created.
docker run –net=host –rm confluentinc/cp-kafka:latest bash -c “seq 42 | kafka-console-producer –broker-list localhost:29092 –topic bar && echo ‘Produced 42 messages.'”
The command above will pass 42 integers using the Console Producer that is shipped with Kafka.
As a result, you should see something like this in your terminal:
Produced 42 messages.
9- It looked like things were successfully written, but let’s try reading the messages back using the Console Consumer and make sure they’re all accounted for.
docker run –net=host –rm confluentinc/cp-kafka:latest kafka-console-consumer –bootstrap-server localhost:29092 –topic bar –new-consumer –from-beginning –max-messages 42
**it might take some time for this command to return data. Kafka has to create the __consumers_offset topic behind the scenes when you consume data for the first time and this may take some time.