2. The message delivery mechanism
1) First zookeeper cluster deployment
# Zookeeper modify configuration files
# Specified node identifier (1,2,3,4 ............)
# Modify the configuration file
# Modify other two kafka profile
1. kafka (Kafka)
Kafka was developed by the Apache Software Foundation, an open source stream processing platform, written by Scala and Java.
kafka is a high throughput of distributed publish-subscribe messaging system, it can handle all the actions of consumers streaming data in the network.
- kafka components
Server Kafka used to live : middleware messaging system, the producer receives the generated message, receiving consumer subscription will be delivered to the consumer consumption process.
producer producer
consumer the consumer, a plurality of groups of consumers are formed
partition partition data is arranged, offset by the offset arrangement, offset to distinguish between data recording, there is a message carrier corresponds to queue
offset offset recorded data partition
zookeeper equivalent exchange
vhost = topic theme, to be careful to classify a type of a theme
[Pictures of foreign chains dumping fails, the source station may have security chain mechanism, it is recommended to save the picture down uploaded directly (img-uimr7l1p-1583405061972) (C: \ Users \ Administrator \ AppData \ Local \ YNote \ data \ weixinobU7VjmoDcK6mV43CyWcqdjkLo5g \ 433ec0fbc6b84d0c8416f47a77543d38} $ e0novktaly1cqei] n97f9.png)]
ZooKeeper : key-value pairs used to store meta information (raw data, the lowest level data) as well as watch discovery mechanism
1.broken node registry: borken registered node generates znode save information temporary node
2.broken topic registry: when a zookeeper will start registration information they hold and the topic of partition
3.consumer and consumer group: mainly used for load balancing
4.consumer id registry: Each consumer has a unique id number for consumer information labeling
5.consumer offset tracking maximum offset is used to track each concumer consumption
6.partition owner registry: used to mark the partition is consumed by the consumer
kafka principle:
https://zhuanlan.zhihu.com/p/68052232
[Pictures of foreign chains dumping fails, the source station may have security chain mechanism, it is recommended to save the picture down uploaded directly (img-P4kGxwNx-1583405061977) (C: \ Users \ Administrator \ AppData \ Local \ YNote \ data \ weixinobU7VjmoDcK6mV43CyWcqdjkLo5g \ 2a82daa6f26f4d25865f4ae754e8af1d \ clipboard.png)]
2. The message delivery mechanism
1) at most once a message is sent at most once, regardless of success or failure, will not send
2) at least once send a message happened
3. kafka advantages:
1) to ensure that everyone-messages, all incoming messages, flew to tag (for example: 1,2,3,4 ......)
2) When the message is consumed, data loss
3) Distributed
4) a relatively large capacity, kafka capacity depends on the size of the hard disk
5) to the size does not affect the performance of kafka
4. kafka cluster
leader: for real work
follower: the information is used to copy the leader, make a backup
lab environment:
server 1： 1.1.1.101/8
server 2： 1.1.1.102/8
server 3： 1.1.1.103/8
1) First zookeeper cluster deployment
kafka 依赖于 zookeeper 来保存集群的数据信息，从而来保证系统的可用性
Three servers, you need to have java environment
- Deploy Java environment, zookeeper dependent on the java environment
- Source code can also be used to install java, details Reference: https://blog.csdn.net/RunzIyy/article/details/104569137
[root@localhost ~]# yum -y install java
[root@localhost ~]# java -version
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)
- Installation zookeeper
[root@localhost ~]# tar -zxf zookeeper-3.3.6.tar.gz -C /usr/src/
[root@localhost ~]# mv /usr/src/zookeeper-3.3.6/ /usr/local/zookeeper
[root@localhost ~]# cd /usr/local/zookeeper/conf/
# 因程序不识别 zoo_sample.cfg 文件，所以需要改名
[root@localhost conf]# cp zoo_sample.cfg zoo.cfg
- Modify the configuration file zookeeper
[root@localhost ~]# vim /usr/local/zookeeper/conf/zoo.cfg
tickTime=2000 # 节点之间发送心跳包的时间， 单位 毫秒
initLimit=10 # 新加入节点初始化的时间， 单位 个 10*2000
syncLimit=5 # 节点连接的超时等待时间
dataDir=/usr/local/zookeeper/data # 指定保存数据的目录
dataLogDir=/usr/local/zookeeper/datalog # 指定数据日志的目录
# the port at which the clients will connect
clientPort=2181 # 对外提供服务器的端口
server.1=1.1.1.101:2888:3888
server.2=1.1.1.102:2888:3888
server.3=1.1.1.103:2888:3888
# 节点名 = 节点IP地址：节点通信的端口：节点之间选取leader的端口
- Create a data storage directory
- Due to changes in the configuration file, specify the path to store data and log file storage location, so the need to manually create
[root@localhost ~]# mkdir /usr/local/zookeeper/data
[root@localhost ~]# mkdir /usr/local/zookeeper/datalog
- Specified node identifier (1,2,3,4 ............)
- zk cluster node needs to obtain myid contents of the file to identify the node, or not start
- The logo similar to the person's identity card number
[root@localhost ~]# echo 1 > /usr/local/zookeeper/data/myid # 指定节点标识 [root@localhost ~]# cat /usr/local/zookeeper/data/myid
1
- For convenience, this program transmission in the past, but the other two need to change the node ID
[root@localhost ~]# scp -r /usr/local/zookeeper/ [email protected]:/usr/local/
[root@localhost ~]# scp -r /usr/local/zookeeper/ [email protected]:/usr/local/
# 主机 2
[root@two ~]# echo 2 > /usr/local/zookeeper/data/myid
[root@two ~]# cat /usr/local/zookeeper/data/myid
2
# 主机 3
[root@three ~]# echo 3 > /usr/local/zookeeper/data/myid
[root@three ~]# cat /usr/local/zookeeper/data/myid
3
- Start zookeeper Service
- Upon completion of the cluster, you need to start the service, has launched three servers to
- start stop start stop restart restart status view status
# 创建软连接，使 启动命令可以在全局中使用
[root@localhost ~]# ln -s /usr/local/zookeeper/bin/* /usr/bin/
[root@localhost ~]# zkServer.sh start
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
- View zookeeper status
- Only all, start to see zookeeper state, will be successful
- When the three servers, all starts, they will be automatically elected between a loader, the rest of foolower (generated random)
zookeeper state
- leader for real work.
- follower information is used to copy the leader, make a backup
# 只有三台 服务器，全部启动 zookeeper服务，才可以查看状态
[root@localhost zookeeper]# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@two ~]# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@three ~]# zkServer.sh status
JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
2) cluster deployment kafka
- First unzip kafka source package, also do not need to compile and install
[root@localhost ~]# tar -zxf kafka_2.11-1.0.1.tgz -C /usr/src/ [root@localhost ~]# mv /usr/src/kafka_2.11-1.0.1/ /usr/local/kafka
- Modify the configuration file
[root@localhost ~]# vim /usr/local/kafka/config/server.properties
21 broker.id=1 # 本机 节点标识
31 listeners=PLAINTEXT://1.1.1.101:9092 # 监听 本地 ip与 端口
60 log.dirs=/usr/local/kafka/data # 指定日志文件的目录
103 log.retention.hours=168
104 message.max.byte=1024000 # 消息最大字节
105 default.replication.factor=2
106 replica.fetch.max.bytes=102400
126 zookeeper.connect=1.1.1.101:2181,1.1.1.102:2181,1.1.1.103:2181 # 指定 zookeeper群集内的节点服务器
107 num.rtition=1
# 创建 日志文件的目录
[root@localhost ~]# mkdir /usr/local/kafka/data
- The other two kafka deployment
- Use scp command to configure the file transfer in the past, modify the configuration file
[root@localhost ~]# scp -r /usr/local/kafka/ [email protected]:/usr/local/ [root@localhost ~]# scp -r /usr/local/kafka/ [email protected]:/usr/local/
- The other two kafka modify configuration files
- Only need to modify, IP address, and identification of the machine monitor
[root@two ~]# vim /usr/local/kafka/config/server.properties
broker.id=2
listeners=PLAINTEXT://1.1.1.102:9092
-------------------------------------------------------
[root@three ~]# vim /usr/local/kafka/config/server.properties
broker.id=2
listeners=PLAINTEXT://1.1.1.102:9092
- Start kafka Service
- It was started three servers
[root@localhost ~]# cd /usr/local/kafka/bin
[root@localhost bin]# ./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@localhost bin]# netstat -anpt | grep 9092
[root@two bin]# cd /usr/local/kafka/bin/
[root@two bin]# ./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@two bin]# netstat -anpt | grep 9092
[root@three ~]# cd /usr/local/kafka/bin/
[root@three bin]# ./kafka-server-start.sh -daemon /usr/local/kafka/config/server.properties
[root@three bin]# netstat -anpt | grep 9092
- Creating topic test
- Use this command to create a message queue
[root@localhost bin]# ./kafka-topics.sh --create --zookeeper 1.1.1.101:2181 --partitions 1 --replication-factor 2 -topic logs
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2020-02-11 19:02:30,208] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
Created topic "logs".
[root@localhost bin]# ./kafka-topics.sh --list --zookeeper 1.1.1.101:2181
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
[2020-02-11 19:02:42,645] WARN Connected to an old server; r-o mode will be unavailable (org.apache.zookeeper.ClientCnxnSocket)
logs
Definition:
# create 创建
# zookeeper 使用哪一个 zookeeper
# -- partition 创建分区的个数
# -- replication-factor 2 指定分区别分的个数
# --toplic 主题名称
- Producers mode
[root@localhost bin]# ./kafka-console-producer.sh --broker-list 1.1.1.102:9092 --topic logs
# --broker-list 1.1.1.102:9092 推送到那个节点上
- Analog consumers
[root@two bin]# ./kafka-console-consumer.sh --zookeeper 1.1.1.101:2181 --topic logs --from-beginning
# --zookeeper 1.1.1.101:2181
# --from-beginning 从哪里读消息，从开始读消息
- This time the producers and consumers will enter the blocked state
@localhost bin] # ./kafka-console-producer.sh --broker-list 1.1.1.102:9092 --topic logs
-broker-list 1.1.1.102:9092 pushed onto that node
- 模拟消费者
```shell
[root@two bin]# ./kafka-console-consumer.sh --zookeeper 1.1.1.101:2181 --topic logs --from-beginning
# --zookeeper 1.1.1.101:2181
# --from-beginning 从哪里读消息，从开始读消息
- This time the producers and consumers will enter the blocked state
- The producer enter anything, the consumer will be displayed in the screen
