Kafka definition, installation, addition, deletion, modification and query commands

1 Overview

Kafka is a distributed message queue (Message Queue) based on the publish/subscribe model , which is mainly used in the real-time processing of big data.

The traditional application areas of message queues are asynchronous processing, traffic peak reduction and decoupling

Two modes of message queue:

Point-to-point mode (one-to-one, consumers actively pull data, and the message is cleared after the message is received)

Publish/subscribe model (one-to-many, messages will not be cleared after consumers consume data)

2. Installation

Modify the configuration file server.propertities after decompression

#broker的全局唯一编号，不能重复，每个节点的id不一样
broker.id=0
#删除topic功能使能
delete.topic.enable=true
#kafka运行日志存放的路径，需要自己创建
log.dirs=/opt/module/kafka/logs
#配置连接Zookeeper集群地址
zookeeper.connect=hadoop102:2181,hadoop103:2181,hadoop104:2181

Configure environment variables

$ sudo vi /etc/profile

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka

$ source /etc/profile

3. Kafak command

Startup and shutdown

#启动
bin/kafka-server-start.sh -daemon config/server.properties
#关闭
bin/kafka-server-stop.sh stop

Add, delete, modify

创建
bin/kafka-topics.sh --zookeeper hadoop102:2181 --create --replication-factor 3 --partitions 
 1 --topic first
    选项说明：
    --topic 定义topic名
    --replication-factor  定义副本数
    --partitions  定义分区数

删除
bin/kafka-topics.sh --zookeeper hadoop102:2181 --delete --topic first
需要server.properties中设置delete.topic.enable=true否则只是标记删除。

查看
    查看所有
    bin/kafka-topics.sh --zookeeper hadoop102:2181 --list
    
    查看某一个topic
    bin/kafka-topics.sh --zookeeper hadoop102:2181 --describe --topic first

修改分区数
bin/kafka-topics.sh --zookeeper hadoop102:2181 --alter --topic first --partitions 6

消费消息
bin/kafka-console-consumer.sh --bootstrap-server hadoop102:9092 --from-beginning --topic first
    --from-beginning：会把主题中以往所有的数据都读取出来。