[docker] docker-compose install kafka cluster

Preface

docker-composeThe network module and use to install the zookeeper cluster were introduced before docker-compose. ( [docker] docker-compose installs the zookeeper cluster )

In this chapter, I will introduce how to use the docker network and installed zk cluster from the previous issue to create your own kafka cluster.

Basic dependencies

docker & docker-compose
zookeeper single node or cluster

The basic idea

Idea 1

Due to the Kafka cluster, there is no official version. According to research, general blogs generally use clusters made by the following two big guys.
Insert image description here
https://hub.docker.com/r/bitnami/kafka

Download the docker image in advance
docker pull bitnami/kafka:2.7.0

PS: For future development and deployment stability, it is recommended to set a more stable version instead of using latestversion, because some contents of each version may be inconsistent.

Others prepare in advance

Create docker network in advance
docker-network create zk-net

Standalone - command line startup

Before deploying using the cluster version, we can refer to the deployment documents provided by the author to conduct deployment testing.

Step 1: Create a network (create docker networks)

$ docker network create app-tier --driver bridge

Step 2: Launch the Zookeeper server instance (deploy the stand-alone version of zookeeper)

$ docker run -d --name zookeeper-server \
    --network app-tier \
    -e ALLOW_ANONYMOUS_LOGIN=yes \
    bitnami/zookeeper:latest

Step 2: Launch the Apache Kafka server instance (deploy the stand-alone version of kafka)

$ docker run -d --name kafka-server \
    --network app-tier \
    -e ALLOW_PLAINTEXT_LISTENER=yes \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
    bitnami/kafka:latest

Step 3: Launch your Apache Kafka client instance (use kafka’s command line tool for testing)

$ docker run -it --rm \
    --network app-tier \
    -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 \
    bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka-server:9092

PS: In this step, docker exec -it <cointainerId> /bin/bashafter entering the kafka container, kafka-topic.shit is also possible to directly call the script for testing.

Standalone - (docker-compose.yml file)

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

The configuration file of the cluster is similar to that of a single machine. Let's parse the configuration file of the single machine first. The above is the official single machine deployment file.

networks

networks/app-tier: This is mainly to declare the related network of docker. It is convenient for use below.
driver:bridge: Using docker networks mode, bridge mode.

services

zookeeper: zookeeper singleton mode deployment, and its related configuration.
kafka: kafla singleton mode deployment, and its related configuration.
kafka/environment: Relevant environment variables.
kafka/ports: Mirror.

Cluster - (docker-compose.yml file)

version: "3"

networks:
  zk-net:
    name: zk-net

services:
  kafka01:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka01
    hostname: kafka01
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka01:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net
  kafka02:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka02
    hostname: kafka02
    ports:
      - '9093:9092'
    environment:
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka02:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net
  kafka03:
    image: 'bitnami/kafka:2.7.0'
    container_name: kafka03
    hostname: kafka03
    ports:
      - '9094:9092'
    environment:
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka03:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181,zoo2:2181,zoo3:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    networks:
      - zk-net

Related practical operations

start updocker-compose up -d
Use kafka-topic.shlinks to related kafka clusters, and use commands to execute: create topics, start producers, start consumers, etc.

Reference

[1]. (hub-docker) bitnami/kafka