版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/sinat_31032177/article/details/86232250
一、安装
下载镜像
docker pull wurstmeister/zookeeper
docker pull wurstmeister/kafka
通过docker-compose启动
docker-compose.yml脚本(zk+kafka版)
version: '2'
services:
zoo1:
image: wurstmeister/zookeeper
restart: unless-stopped
hostname: zoo1
ports:
- "2182:2181"
container_name: zookeeper
kafka1:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: localhost
KAFKA_ZOOKEEPER_CONNECT: "zoo1:2182"
KAFKA_BROKER_ID: 1
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_CREATE_TOPICS: "stream-in:1:1,stream-out:1:1"
depends_on:
- zoo1
container_name: kafka
启动
docker-compose up -d
二、使用
进入kafka
docker exec -it kafka /bin/bash
cd /opt/kafka_2.11-1.1.0
创建主题
bin/kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic mykafka //zk为容器版
bin/kafka-topics.sh --create --zookeeper 192.168.247.131:2181 --replication-factor 1 --partitions 1 --topic mykafka //zk为本地版
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mykafka //运行一个消息生产者,指定topic为刚刚创建的主题
bin/kafka-console-consumer.sh --zookeeper 192.168.247.131:2181 --topic mykafka --from-beginning //运行一个消费者,指定同样的主题
查看
bin/kafka-topics.sh --list --zookeeper localhost:2181
发送消息
发送消息
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
启动消费者
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
单节点集群
cp config/server.properties config/server-1.properties
cp config/server.properties config/server-2.properties
config/server-1.properties:
broker.id=2
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-1
zookeeper.connect=192.168.247.131:2182
config/server-2.properties:
broker.id=3
listeners=PLAINTEXT://:9094
log.dirs=/tmp/kafka-logs-2
zookeeper.connect=192.168.247.131:2183
bin/kafka-server-start.sh config/server-1.properties &
bin/kafka-server-start.sh config/server-2.properties &
//配置并启动基于zk集群的两个kafka
bin/kafka-topics.sh --create --zookeeper 192.168.247.131:2182 --replication-factor 2 --partitions 1 --topic my-replicated-topic
复制数量小于等于相关联的zk集群数,且端口号必须是集群内的zk
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
//查看集群状态
//“leader”是负责给定分区的所有读取和写入的节点。每个节点将成为随机选择的分区部分的领导者。
//“replicas”是复制此分区日志的节点列表,无论它们是否为领导者,或者即使它们当前处于活动状态。
//“isr”是“同步”复制品的集合。这是副本列表的子集,该列表当前处于活跃状态并且已经被领导者捕获。
数据处理(使用Kafka Connect导入/导出数据)
echo -e "foo\nbar" > test.txt
bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-file-sink.properties
more test.sink.txt
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
echo Another line>> test.txt