版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/u012292754/article/details/84782875
1 Kafka概述
和消息系统类似
1.1 消息中间件
生产者和消费者
1.2 Kafka 架构和概念
- producer:生产者(生产馒头)
- consumer:消费者(吃馒头)
- broker:篮子
- topic : 主题,给馒头带一个标签,(topic-A 给A吃,topic-B 给B吃)
2 Kafka 部署及使用(单节点单broker)
- 单节点单 Broker
- 单节点多 Broker
- 多节点多 Broker
2.1 需要先安装 Zookeeper
http://archive-primary.cloudera.com/cdh5/cdh/5/
配置到环境变量,然后刷新
export ZK_HOME=/home/hadoop/apps/zookeeper-3.4.5-cdh5.7.0
export PATH=$PATH:$ZK_HOME/bin
修改 zoo.cfg
启动测试
2.2 安装 Kafka
下载https://kafka.apache.org/downloads
解压,配置到环境变量
export KAFKA_HOME=/home/hadoop/apps/kafka_2.11-0.9.0.0
export PATH=$PATH:$KAFKA_HOME/bin
2.3 配置Kafka
/home/hadoop/apps/kafka_2.11-0.9.0.0/config
server.properties
host.name=node1
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs
zookeeper.connect=node1:2181
2.4 测试 Kafka
前提首先启动 zookeeper,zkServer.sh start
kafka-server-start.sh $KAFKA_HOME/config/server.properties
创建 topic:指定zk
[hadoop@node1 ~]$ kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 1 --partitions 1 --topic hello_tpoic
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "hello_tpoic".
[hadoop@node1 ~]$ kafka-topics.sh --list --zookeeper node1:2181
hello_tpoic
[hadoop@node1 ~]$
发送消息:指定broker
kafka-console-producer.sh --broker-list node1:9092 --topic hello_topic
启动消费者:指定zk
kafka-console-consumer.sh --zookeeper node1:2181 --topic hello_topic --from-beginning
查看信息
[hadoop@node1 ~]$ kafka-topics.sh --describe --zookeeper node1:2181
Topic:hello_topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:hello_tpoic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello_tpoic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
[hadoop@node1 ~]$ kafka-topics.sh --describe --zookeeper node1:2181 --topic hello_topic
Topic:hello_topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
[hadoop@node1 ~]$
3 单节点多 broker 的部署
[hadoop@node1 config]$ cp server.properties server-1.properties
[hadoop@node1 config]$ cp server.properties server-2.properties
[hadoop@node1 config]$ cp server.properties server-3.properties
3.1 修改配置文件
server-1.properties
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-1
server-2.properties
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-2
server-3.properties
broker.id=3
listeners=PLAINTEXT://:9095
log.dirs=/home/hadoop/appsData/kafkaData/kafka-logs-3
3.2 启动
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &
kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &
创建 topic
kafka-topics.sh --create --zookeeper node1:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
[hadoop@node1 config]$ kafka-topics.sh --list --zookeeper node1:2181
hello_topic
hello_tpoic
my-replicated-topic
[hadoop@node1 config]$ kafka-topics.sh --describe --zookeeper node1:2181
Topic:hello_topic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:hello_tpoic PartitionCount:1 ReplicationFactor:1 Configs:
Topic: hello_tpoic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 3 Replicas: 3,2,1 Isr: 3,2,1
[hadoop@node1 config]$
创建生产者
kafka-console-producer.sh --broker-list node1:9093,node1:9094,node1:9095 --topic my-replicated-topic
创建消费者
kafka-console-consumer.sh --zookeeper node1:2181 --topic my-replicated-topic