[Yugong Series] Construction of Docker Container Kafka Cluster in March 2022

---

First, the construction of Kafka cluster

1. Pull related images

docker pull wurstmeister/kafka
docker pull zookeeper

2. Run zookeeper

docker run -d --name zookeeper -p 2181:2181 -t zookeeper

3. Run kafka

Kafka0：

docker run -d --name kafka0 -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.16.129:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.16.129:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka

Kafka1 ：

docker run -d --name kafka1 -p 9093:9093 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT=192.168.16.129:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.16.129:9093 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9093 -t wurstmeister/kafka

Kafka2：

docker run -d --name kafka2 -p 9094:9094 -e KAFKA_BROKER_ID=2 -e KAFKA_ZOOKEEPER_CONNECT=192.168.16.129:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.16.129:9094 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9094 -t wurstmeister/kafka

Parameter Description:

• -e KAFKA_BROKER_ID=0In a kafka cluster, each kafka has a BROKER_ID to distinguish itself
• -e KAFKA_ZOOKEEPER_CONNECT=10.20.8.50:2181/kafkaConfigure zookeeper to manage kafka's path 10.20.8.50:2181/kafka
• -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.20.8.50:9092Register the address port of kafka to zookeeper, if it is remote access, change it to the external network IP.
• -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092Configure the listening port of kafka: this cannot be changed
• -v /etc/localtime:/etc/localtimeThe container time synchronizes the time of the virtual machine

Start 3 Kafka nodes

4. Set topic

enter kafka0

docker exec -it kafka0 /bin/bash

Enter the bin directory

cd /opt/kafka_2.13-2.8.1/bin

create topic

kafka-topics.sh --create --zookeeper 192.168.16.129:2181 --replication-factor 3 --partitions 5 --topic TestTopic

View topic

kafka-topics.sh --describe --zookeeper 192.168.16.129:2181 --topic TestTopic

All partitions of Kafka's topic will be distributed on different brokers, so the 5 partitions of the topic will be distributed to 3 brokers, of which two brokers get two partitions, and the other broker has only one partition, as shown in the figure :

Cluster node description:

• Topic: TestTopic PartitionCount: 5 ReplicationFactor:3It means that TestTopic has 5 partitions and 3 replica nodes;
• Topic: represents the subject name
• Leaderrepresents the topic node number,
• ReplicasRepresenting his replica node has Broker.id = 2, 0, 1 (including Leader Replica and Follower Replica, regardless of whether it is alive or not),
• IsrIndicates that the replica that is alive and synchronized with the Leader node has Broker.id = 2, 0, 1

5. Conduct producer and consumer tests

Run a producer on Broker0 and a consumer on Broker1 and Broker2 respectively:

kafka-console-producer.sh --broker-list 192.168.16.129:9092 --topic TestTopic

kafka-console-consumer.sh --bootstrap-server 192.168.16.129:9093 --topic TestTopic --from-beginning

kafka-console-consumer.sh --bootstrap-server 192.168.16.129:9094 --topic TestTopic --from-beginning