基于Docker的Kafka部署

一 准备

1.1 安装docker-dompose

#部署
sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
#设置权限
chmod +x /usr/bin/docker-compose
#验证
docker-compose version

　　

1.2 下载镜像

docker pull wurstmeister/zookeeper:latest
docker pull wurstmeister/kafka:0.10.0.1

　　

也可以使用Dockerfile构建，构建文件都在Github上https://github.com/jdlzy/kafka-docker

二 部署Zookeeper

首先需要新建专用网络

docker network create --driver bridge --subnet 172.23.0.0/25 --gateway 172.23.0.1 zookeeper_network

部署zookeeper需要在任意目录下新建docker-compose.yml，将下边的内容拷贝到目录中。

#前台启动
docker-compose up
#后台启动
docker-compose up -d

2.1 yml文件

version: '3.4'
services:
  zoo1:
    image: wurstmeister/zookeeper
    restart: always
    hostname: zookeeper
    container_name: zookeeper
    ports:
    - 2181:2181
    volumes:
    - "/root/lzy/kafka-docker/data/kafka/data:/data"
    - "/root/lzy/kafka-docker/data/kafka/datalog:/datalog"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zookeeper:2888:3888
    networks:
      default:
        ipv4_address: 172.23.0.11
networks:
  default:
    external:
      name: zookeeper_network

三 Kafka部署

在任意目录下新建docker-compose.yml，将下边的内容拷贝到目录中。

#前台启动
docker-compose up
#后台启动
docker-compose up -d

3.1 yml文件

10.95.3.172是我的物理机的IP，这里穿件3个节点，Broker端口分别为9092、9093、9094

version: '2'
services:
  kafka1:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka1
    container_name: kafka1
    ports:
    - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9092
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.14
  kafka2:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka2
    container_name: kafka2
    ports:
    - "9093:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9093 
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:  # 连接本compose文件以外的container
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.15
  kafka3:
    image: wurstmeister/kafka:0.10.0.1
    restart: always
    hostname: kafka3
    container_name: kafka3
    ports:
    - "9094:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://10.95.3.172:9094
      KAFKA_ADVERTISED_HOST_NAME: 10.95.3.172
      KAFKA_ADVERTISED_PORT: 9094
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    external_links:  # 连接本compose文件以外的container
    - zookeeper
    networks:
      default:
        ipv4_address: 172.23.0.16
networks:
  default:
    external:   # 使用已创建的网络
      name: zookeeper_network

四 测试

在任意物理机（需要部署了wurstmeister/kafka:0.10.0.1镜像），或者直接下载0.10.0.1的Kafka安装包。 
如果是基于Docker，执行下边的命令启动一个临时容器

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka:0.10.0.1 /bin/bash

如果是基于Kafka安装包，直接解压即可。 

然后到/opt/kafka/bin目录下

4.1 创建Topic

–replication-factor表示副本数，–partitions表示分片数量

./kafka-topics.sh --create --zookeeper 10.95.3.172:2181 --replication-factor 2 --partitions 3 --topic t1
#查看topic
./kafka-topics.sh --list --zookeeper 10.95.3.172:2181

4.2 创建生产者

./kafka-console-producer.sh --broker-list 10.95.3.172:9092,10.95.3.172:9093,10.95.3.172:9094 --topic t1

4.3 创建消费者

–from-beginning表示从消息起始开始读取

./kafka-console-consumer.sh --zookeeper 10.95.3.172:2181 --topic t1 --from-beginning

五 非docker-compose部署方式

不使用docker-compose可以进行分布式部署，但是目前还有问题，有待进一步测试

5.1 部署zookeeper

docker run -d --name zookeeper --publish 2181:2181 --volume /etc/localtime:/etc/localtime wurstmeister/zookeeper:latest

5.2 部署kafka

创建多节点时候需要修改KAFKA_BROKER_ID为唯一， 
-p映射到物理机的端口不能重复，第二个可以为9093，第三个可以为9094等。

docker run -d --name kafka \
-e HOST_IP=10.95.3.172 \
-e KAFKA_ADVERTISED_PORT=9092 \
--env KAFKA_ADVERTISED_HOST_NAME=10.95.3.172 \
-e KAFKA_BROKER_ID=1 \
-e ZK=zk \
-p 9092:9092 \
--link zookeeper:zk \
-t wurstmeister/kafka:0.10.0.1

 转载：http://blog.leanote.com/archives/kobeliuziyang