Use docker to build kafka environment under Mac

Since kafka depends on zookeeper, you need to use docker to install zookeeper and kafka at the same time.

x01 install zookeeper

1) Download the zookeeper image first

> docker search zookeeper
> docker pull wurstmeister/zookeeper

2) Start the zookeeper service

> docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2  --name zookeeper -p 2181:2181 -v /etc/localtime:/etc/localtime wurstmeister/zookeeper

x02 install kafka

1) Download the kafka image first

> docker search kafka
> docker pull wurstmeister/kafka

2) Start the kafka service

> docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name kafka -p 9092:9092 -e KAFKA_BROKER_ID=0 -e KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181/kafka -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9092 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -v /etc/localtime:/etc/localtime wurstmeister/kafka

x03 message verification

1) Enter the kakfa container

> dcker exec -it kafka bash
> cd /opt/kafka/bin

You can see a lot of scripts:

> ls
connect-distributed.sh               kafka-console-producer.sh            kafka-leader-election.sh             kafka-run-class.sh                   trogdor.sh
connect-mirror-maker.sh              kafka-consumer-groups.sh             kafka-log-dirs.sh                    kafka-server-start.sh                windows
connect-standalone.sh                kafka-consumer-perf-test.sh          kafka-mirror-maker.sh                kafka-server-stop.sh                 zookeeper-security-migration.sh
kafka-acls.sh                        kafka-delegation-tokens.sh           kafka-preferred-replica-election.sh  kafka-streams-application-reset.sh   zookeeper-server-start.sh
kafka-broker-api-versions.sh         kafka-delete-records.sh              kafka-producer-perf-test.sh          kafka-topics.sh                      zookeeper-server-stop.sh
kafka-configs.sh                     kafka-dump-log.sh                    kafka-reassign-partitions.sh         kafka-verifiable-consumer.sh         zookeeper-shell.sh
kafka-console-consumer.sh            kafka-features.sh                    kafka-replica-verification.sh        kafka-verifiable-producer.sh

2) Test producer and consumer
Producer

> ./kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic

Consumer (requires another terminal)

./kafka-console-consumer.sh --bootstrap-server host.docker.internal:9092 --topic test-topic --from-beginning

Note ⚠️:
When registering with zookeeper here, host.docker.internal is used. When we connect to kafka in the program, using localhost directly will report an error, such as: Error connecting to node host.docker.internal:9092, and its processing Way:

• You can add a mapping in the host file of this machine, and set 127.0.0.1 host.docker.internal;
• Do not use host.docker.internal, but use your own host IP, but when the IP changes, you need to reconfigure.

x04 kafka cluster mode construction

Create a new kafka instance:

> docker run -d --restart=always --log-driver json-file --log-opt max-size=100m --log-opt max-file=2 --name kafka -p 9093:9093 -e KAFKA_BROKER_ID=1 -e KAFKA_ZOOKEEPER_CONNECT=host.docker.internal:2181/kafka -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9093 -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9093 -v /etc/localtime:/etc/localtime wurstmeister/kafka

1. Change broker.id: KAFKA_BROKER_ID 0, 1, 2
2. Change port numbers: 9092, 9093, 9094

x05 kafka-manger installation

1) install

> docker pull sheepkiller/kafka-manager

2) start

docker run -d --name kfk-manager --restart always -p 9000:9000 -e ZK_HOSTS=<这里换成你的zookeeper地址和端口> sheepkiller/kafka-manager

3) Access the address localhost:9000

to configure your own zookeeper address;
Note ⚠️:
I initially configured localhost:2181, and found that I could not query my own broker information; then I went to zookeeper to check, and I needed to add the path
localhost: 2181/kafka will do.