Deploy kafka with docker

1. Reference link

1. Docker quickly builds Kafka 1.x cluster
2. Kafka Docker - Run multiple Kafka brokers in Docker
3. Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node
4. Using Vagrant to get up and running with Apache Kafka
5. The difference and application of kafka listeners and advertised.listeners
6. Kafka configures KAFKA_LISTENERS and KAFKA_ADVERTISED_LISTENERS

2. Excerpt Citation

2.1 Docker image selection

Kafka does not have an official Docker image, so either write a Dockerfile yourself, or use a third-party built one.

It is not impossible to write a Dockerfile by yourself, but it does not meet my "fastest" goal, so it is the fastest to use a third-party built image.

Since it is a third-party mirror, I hope that the more people who have used it, the better, so that there will be relatively fewer pitfalls.

With the above requirements in mind, we began to look for suitable third-party mirrors. The more famous ones are as follows:

1. wurstmeister/kafkaFeatures: The number of stars is the largest, the version is updated to Kafka 1.0, and zookeeper and kafka are separated from different mirrors.
2. spotify/kafkaFeatures: The number of stars is large, and many articles or tutorials recommend it. Zookeeper and Kafka are placed in the same image; but the version of Kafka is relatively old (still at 0.10.1.0).
3. confluent/kafkaBackground: Confluent is a new company founded by Jay Kreps, who developed Kafka mentioned in the book, after leaving LinkedIn. Confluent Platform is a streaming data platform that has built a series of products around Kafka. Features: It is written by a big guy, with detailed documentation, but it is also bundled with Confluent Platform.

Among the above three projects, the one that was finally selected for use wurstmeister/kafkahas the most stars, and the version has been kept updated, so it should be more reassuring to use.

2.2 wurstmeister/kafkaPrerequisites for Introduction to Mirroring

• Installdocker-compose
• Modify in docker-compose.ymlthe configuration file KAFKA_ADVERTISED_HOST_NAMEto match your docker host IP (note: if you want to run a multi-broker cluster, do not use localhostor 127.0.0.1as the host ip)
• If you want to add custom Kafka parameters, just add them to docker-compose.yml. For example:
  • Add message.max.bytesparameter, add KAFKA_MESSAGE_MAX_BYTES: 2000000to environmentsection.
  • Turn off automatic topic creation set, configureKAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
• LOG4J_Kafka can customize log4j usage by prefixing environment variables . These environment variables are mapped to log4j.properties. For example:LOG4J_LOGGER_KAFKA-AUTHORIZER_LOGGER=DEBUG,authorizerAppender

**NOTE:** There are several "problems" with configuring the network. If you don't know what the requirements are, check out the hookup guide

2.3 Easy to use

Start the cluster:

• docker-compose up -d

Add more brokers

• docker-compose scale kafka=3

destroy a cluster

• docker-compose stop

3. Installation process

3.1 Install Docker (omitted)

3.2 Install Docker Compose (omitted)

3.3 Pull the latest packaged resources from github

# 克隆docker制作脚本
git clone https://github.com/wurstmeister/kafka-docker.git

# 进入目录查看目录文件
cd kafka-docker

# 查看所有标签，找到最新版本
git tag

# 切换到最新版本
git checkout <last_tag_name>

Kafka-docker warehouse directory file display:

[root@jiewli kafka-docker]# ll
总用量 72
-rwxr-xr-x. 1 root root   210 7月   7 17:18 broker-list.sh
-rw-r--r--. 1 root root   969 7月   7 17:18 CHANGELOG.md
-rwxr-xr-x. 1 root root  1221 7月   7 17:18 create-topics.sh
-rw-r--r--. 1 root root   367 7月   7 17:08 docker-compose-single-broker.yml
-rw-r--r--. 1 root root   705 7月   7 17:08 docker-compose-swarm.yml
-rw-r--r--. 1 root root   324 7月   7 17:18 docker-compose.yml
-rw-r--r--. 1 root root  1132 7月   7 17:18 Dockerfile
-rwxr-xr-x. 1 root root   395 7月   7 17:18 download-kafka.sh
-rw-r--r--. 1 root root 11325 7月   7 17:08 LICENSE
-rw-r--r--. 1 root root  9887 7月   7 17:18 README.md
-rwxr-xr-x. 1 root root  4591 7月   7 17:18 start-kafka.sh
-rwxr-xr-x. 1 root root   131 7月   7 17:08 start-kafka-shell.sh
drwxr-xr-x. 2 root root  4096 7月   7 17:18 test

3.4 Update docker-compose.ymlyour docker host ip

Use the command vim docker-compose.ymlto view the configuration files in the warehouse directory docker-compose.yml.

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092"
    environment:
      DOCKER_API_VERSION: 1.22
      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Important:
Versions after kafka 0.9.x add a new advertised.listenersconfiguration.
Versions after kafka 0.9.x do not use advertised.host.nameand advertised.host.porthave been deprecated
host.nameand portare deprecated, use listenersinstead

Therefore, KAFKA_ADVERTISED_HOST_NAMEthe configuration should also be changed to KAFKA_ADVERTISED_LISTENERS, and the latter requires configuration KAFKA_LISTENERS, otherwise an exception will be thrown:

ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092"
    environment:
      DOCKER_API_VERSION: 1.22
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.99.100:9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Change the above KAFKA_ADVERTISED_HOST_NAMEto your docker host ip.

If you want to add custom Kafka parameters, just add them to docker-compose.yml. For example:

• Add message.max.bytesparameter, add KAFKA_MESSAGE_MAX_BYTES: 2000000to environmentsection.
• Turn off automatic topic creation set, configureKAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

Startup failed:

[root@jiewli kafka-docker]# docker logs -f --tail=300 kafka-docker_kafka_1
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS
ERROR: Missing environment variable KAFKA_LISTENERS. Must be specified when using KAFKA_ADVERTISED_LISTENERS

3.5 Start the cluster

$ docker-compose up -d

For example, to start a brokercluster with two

$ docker-compose scale kafka=2

This starts a single zookeeperinstance, and two Kafkainstances. You can use docker-compose psthe command to display running instances.
If you want to add more Kafkabrokers, just docker-compose scale kafka=nincrease the value through the command.

docker-compose -f docker-compose.yml up -d3.5.1 Error encountered when executing command

wget: server returned error: HTTP/1.1 404 Not Found
The command '/bin/sh -c apk add --no-cache bash curl jq docker  && mkdir /opt  && chmod a+x /tmp/*.sh  && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /usr/bin  && sync && /tmp/download-kafka.sh  && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt  && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz  && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka  && rm /tmp/*  && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk  && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk  && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 1
ERROR: Service 'kafka' failed to build

According to the prompt, wget404 was encountered when the command downloaded a file.

3.5.2 Analyzing script execution process

From the Dockerfileinstructions RUNyou can find:

After some investigation, I finally located the command download-kafka.shin this script . I added a line of command and printed the whole command to see:wgetechowget

# echo打印结果
wget -q "https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz" -O "/tmp/kafka_2.12-1.1.0.tgz"

Find this file according to the link, and found that it was downloaded from the "Tsinghua University Open Source Software Mirror Station", but 1.1.0this directory is gone, replaced by 2.x.xthe directory.

Can't download, obviously this version is too behind. It turns out that the warehouse git-tag was created in 2018, and no new tag version has been released for more than three years.

3.5.3 Use the build script of the master branch to start 2.x.xthe version of kafka

Then go back to masterthe branch and look at the latest submission, which was submitted in June 2021. After carefully studying the script and Dockerfile of the master branch, the author has already adjusted the kafka version to 2.x.x, and modified the download address.

Then after switching to the master branch, re-use the command $ docker-compose up -dto start the kafka cluster.

3.6 Start a single node

docker-compose -f docker-compose-single-broker.yml up -d

Opening the container looks like this:

[root@jiewli ~]# docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS       PORTS                                                NAMES
43fb3efd832f   kafka-docker_kafka          "start-kafka.sh"         24 hours ago    Up 4 hours   0.0.0.0:9092->9092/tcp                               kafka-docker_kafka_1
b8bc06090259   wurstmeister/zookeeper      "/bin/sh -c '/usr/sb…"   24 hours ago    Up 4 hours   22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka-docker_zookeeper_1

4. Kafka Shell

Enter the docker container interactive command line

docker exec -it kafka-docker_kafka_1 /bin/bash

Check the kafka installation directory and find the script location

bash-5.1# echo $KAFKA_HOME
/opt/kafka
bash-5.1# cd $KAFKA_HOME
bash-5.1# ls
LICENSE    NOTICE     bin        config     libs       licenses   logs       site-docs

By default, the kafka installation directory is/opt/kafka

You can interact with your Kafka cluster through the Kafka Shell.

$ $KAFKA_HOME/bin/start-kafka-shell.sh <DOCKER_HOST_IP> <ZK_HOST:ZK_PORT>

5. Test

Test your installation steps.

Start a shell terminal, create a topic and start a producer.

$ $KAFKA_HOME/bin/kafka-topics.sh --create --topic topic \
--partitions 4 --zookeeper $ZK --replication-factor 2

$ $KAFKA_HOME/bin/kafka-topics.sh --describe --topic topic --zookeeper $ZK

$ $KAFKA_HOME/bin/kafka-console-producer.sh --topic=topic \
--broker-list=`broker-list.sh`

Start another shell terminal, and start a consumer.

$ $KAFKA_HOME/bin/kafka-console-consumer.sh --topic=topic --zookeeper=$ZK

6. Run Kafka-docker on a MAC computer

Install Docker Toolboxand docker-machine ipconfigure the ip returned by the command toKAFKA_ADVERTISED_HOST_NAME

7. Troubleshooting

• By default, the Kafka broker uses 1GB of memory, so if you have problems starting the broker, check docker-compose logs/ docker logsfind the container and make sure there is enough free memory on the host.
• If you want to run multiple brokers, please do not use localhostor 127.0.0.1as the host IP, otherwise the brokers will not be able to communicate.