Preface
In the previous section [Kafka Advanced Series 001] Kafka stand-alone installation and startup (Mac) , we demonstrated how to install and start Kafka locally and the production and sending process of Kafka messages. This section will demonstrate how to install and start Kafka containers in Docker. , And how to test the production and sending of messages in the Kafka container.
Note: Docker is an open source application container engine that allows developers to package their applications and dependencies into a lightweight, portable container, and then publish to any popular Linux machine, which can also be virtualized. See the official website: https://www.docker.com/
Learning Docker container technology can easily and flexibly install applications, build environments, and deploy applications. If capable students are advised to get started with Docker early, of course, students who find it difficult for the time being can also ignore the learning in this section, and it will not affect the follow-up learning of Kafka.
1. Install Docker, ZK, Kafka
1. Docker installation
First confirm the local installation of Docker and Docker client, this step will not be detailed in this article.
The Docker client can be downloaded from the download address provided by Ali Mirror (Mac): http://mirrors.aliyun.com/docker-toolbox/mac/docker-for-mac/stable/
The download addresses corresponding to other operating systems can be found in Alibaba Cloud: https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors
brew cask install dockerOne-click installation is available in Mac .
2. Installation of ZK and Kafka
In Docker, you can use commands to search for ZK and Kafka mirrors:
docker search kafkawithdocker search zookeeper
We choose to download the top-ranked mirror "wurstmeister/zookeeper", the command is as follows:
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
After the download is complete, use it docker imagesto verify whether the ZK and Kafka images are downloaded successfully:
Second, use Docker to start Kafka
1. Start ZK
docker run -d --name zookeeper -p 2181:2181 -t wurstmeister/zookeeper
2. Start Kafka
docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.0.100:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.0.100:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka
Note: The IP address after the parameter KAFKA_ZOOKEEPER_CONNECTand KAFKA_ADVERTISED_LISTENERSmust be replaced with the host's IP address, otherwise it will not be able to monitor. The IP address of the host computer can be checked through ifconfig en0(en0 represents the wireless network LAN IP):
3. Verify whether ZK and Kafka are started successfully
docker ps -aCheck whether ZK and Kafka are started successfully through commands , mainly check the STATUS column.
'UP 40 minutes' means that it has been started for 40 minutes. If it is'EXITED(1)', it means that it exited abnormally and did not start successfully. You can docker logs [CONTAINED ID]check the cause of the error by command .
CONTAINER ID represents the container ID that was started, which will be used later. For example, the Kafka container ID started this time is: 629c0f149e27.
Three, Kafka message production and sending in Docker
1. Enter the Kafka container (629c0f149e27 is the Kafka container ID, which is different each time):
docker exec -it 629c0f149e27 /bin/bash
2. Enter the Kafka bin directory:
cd /opt/kafka_2.12-2.4.0/bin
3. Create a producer, the copy is 1, the partition is 1, and the topic is TestTopicForDocker:
kafka-topics.sh --create --zookeeper 192.168.0.100:2181 --replication-factor 1 --partitions 1 --topic TestTopicForDocker
Note: The 192.168.0.100IP needs to be replaced with the corresponding host IP, the same as below.
4. View the newly generated topic:
kafka-topics.sh --zookeeper 192.168.0.100:2181 --list；
5. Run a producer in the bin directory
kafka-console-producer.sh --broker-list 192.168.0.100:9092 --topic TestTopicForDocker
6. In the bin directory, run a consumer:
Open a new window and re-enter the bin directory of the Kafka container:
docker exec -it 629c0f149e27 /bin/bash
cd /opt/kafka_2.12-2.4.0/bin
Create consumers:
kafka-console-consumer.sh --bootstrap-server 192.168.0.100:9092 --topic TestTopicForDocker --from-beginning
If there is no error, it means there is basically no problem and you can test.
7. Test
(1) Open a single producer and a single consumer
Send any content in the producer window, and the consumer window can receive the message, indicating that Kafka has successfully received it.
(2) Open a single producer + multiple consumers
Re-open a command line window and run consumer 2. After the producer sends the message, both consumers 1 and 2 can receive it
(3) Create topic 02, and run a producer and consumer respectively for testing:
Note:
(1) During the test, there have been many times when the kafka container was commanded through docker, the corresponding STATUS immediately entered the Exited(1)state. As mentioned above, this status is due to the abnormal exit of the container startup. After investigation, it is because the host IP address is not modified when Kafka is started, and the ZK cannot be connected, so you must ifconfig en0confirm it through ;
Most of the problems are caused by the IP address parameter when starting Kafka:
docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.1.199:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.199:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka
KAFKA_ZOOKEEPER_CONNECT=192.168.1.199:2181And KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.199:9092these two parameters must change the IP to your own LAN IP.
(2) An Error while fetching metadata with correlation id 2 : {MyTestTopicForDocker=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)abnormality occurred during the test . After inspection, it is caused by incorrect Kafka configuration parameters. It is necessary to confirm that Kafka's server.properties file lacks two parameters listeners=xxand advertised.listeners=causes it. It is necessary to confirm the normal configuration of the parameters.
The location of the server.properties file:/opt/kafka_2.12-2.4.0/config/
