docker kafka single node
Host IP: 192.168.43.177
1. Download the image
used here wurstmeister / kafka and wurstmeister / zookeeper two mirrored versions
docker pull wurstmeister / zookeeper
docker pull wurstmeister / kafka
run docker Images verification image has been installed in the two command
2. Start
Start zookeeper container
docker run -d --name zookeeper -p 2181: 2181 -t wurstmeister / zookeeper
启动kafka容器
docker run -d --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=192.168.43.177:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.43.177:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -t wurstmeister/kafka
3. Go to the container operations:
cd /opt/kafka_2.12-2.3.0/bin
create a theme
sh kafka-topics.sh --create --zookeeper 192.168.43.177:2181 --replication-factor 1 --partitions 2 - topic test
delete topics [in server.properties the need to set delete.topic.enable = true]
sh kafka-topics.sh --delete --zookeeper 192.168.43.177:2181 --topic test
View topic:
SH kafka-topics.sh --zookeeper 192.168.43.177:2181 --list
SH kafka-topics.sh --describe --zookeeper 192.168.43.177:2181 --topic Test
production message:
SH List kafka-console-producer.sh --broker-192.168. 43.177: 9092 --topic test
consumer message:
SH kafka-console-consumer.sh --bootstrap-Server 192.168.43.177:9092 the --from Beginning --topic the Test-
SH Kafka used to live-CONSOL e-consumer.sh --bootstrap- server 192.168.43.177:9092 --topic test
--------------------------------------------------------------------------------
docker kafka cluster deployment
Host IP: 192.168.43.177
1. Download the image
used here wurstmeister / kafka zookeeper and the two versions of the image
Docker pull zookeeper
Docker pull wurstmeister / kafka
run docker images verify that the two mirrors have been installed in the command
2. Write docker-kafka.yml documents, content as follows:
version: '2'
services:
zoo1:
image: zookeeper
hostname: zoo1
container_name: zoo1
ports:
- 2181:2181
environment:
ZOO_MY_ID: 1
ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo2:
image: zookeeper
hostname: zoo2
container_name: zoo2
ports:
- 2182:2181
environment:
ZOO_MY_ID: 2
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181
zoo3:
image: zookeeper
hostname: zoo3
container_name: zoo3
ports:
- 2183:2181
environment:
ZOO_MY_ID: 3
ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181
kafka1:
image: wurstmeister/kafka
hostname: kafka1
container_name: kafka1
ports:
- "9091:9091"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 1
KAFKA_HOST_NAME: kafka1
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka1:9091
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9091
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3
kafka2:
image: wurstmeister/kafka
hostname: kafka2
container_name: kafka2
ports:
- "9092:9092"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 2
KAFKA_HOST_NAME: kafka2
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka2:9092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3
kafka3:
image: wurstmeister/kafka
hostname: kafka3
container_name: kafka3
ports:
- "9093:9093"
depends_on:
- zoo1
- zoo2
- zoo3
environment:
KAFKA_BROKER_ID: 3
KAFKA_HOST_NAME: kafka3
KAFKA_ZOOKEEPER_CONNECT: zoo1:2181,zoo2:2181,zoo3:2181
KAFKA_LISTENERS: PLAINTEXT://kafka3:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9093
volumes:
- /var/run/docker.sock:/var/run/docker.sock
links:
- zoo1
- zoo2
- zoo3
3. Create kafka cluster, the next host docker-kafka.yml file directory
docker-compose -f docker-kafka.yml up -d
4. Modify the host or other client host hosts configuration, the following configuration
192.168.43.177 kafka1
192.168.43.177 kafka2
192.168.43.177 kafka3
Related command:
stop the container:
Docker STOP zoo1 zoo2 zoo3 kafka1 kafka2 kafka3
delete container:
Docker RM zoo1 zoo2 zoo3 kafka1 kafka2 kafka3
create a theme:
SH kafka-topics.sh --create --zookeeper 192.168.43.177:2181,192.168.43.177: 2182,192.168.43.177: 2183 --replication-factor 1 --partitions 2 --topic test
View topic:
SH kafka-topics.sh --list --zookeeper 192.168.43.177:2181,192.168.43.177:2182,192.168. 43.177: 2183
SH kafka-topics.sh --describe --zookeeper 192.168.43.177:2181,192.168.43.177:2182,192.168.43.177:2183 --topic the Test
production message:
SH kafka-console-producer.sh --broker -list 192.168.43.177:9091,192.168.43.177:9092,192.168.43.177:9093 --topic test
consumer message:
sh kafka-console-consumer.sh --bootstrap-server 192.168.43.177:9091,192.168.43.177:9092,192.168.43.177:9093 --from-beginning --topic test
Problem solving:
1, the use of docker-compse run compose-docker.yml encounter problems in windows. Mount denied: \ nThe source path " \\\\ var \\\\ run \\\\ docker.sock: /var/run/docker.sock" \ nis not a valid Windows path
Workaround: Set the environment variable COMPOSE_CONVERT_WINDOWS_PATHS = 1
2, java.io.IOException: Can not resolve address : kafka1: 9091
Solution: Modify the host or other client host hosts configuration, the following configuration
3、[2019-12-04 17:37:00,848] WARN [Producer clientId=console-producer] Received invalid metadata error in produce request on partition test-0 due to org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.. Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
[2019-12-04 17:37:00,953] ERROR Error when sending message to topic test with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
解决方法：修改容器内kafka相关端口，kafka1示例：
ports:
- "9091:9091"
KAFKA_LISTENERS: PLAINTEXT://kafka1:9091
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9091