[Distributed] zookeeper cluster and kafka cluster

Zookeeper cluster + Kafka cluster

1. Overview of Zookeeper

1.1 Zookeeper definition

Zookeeper is an open source distributed Apache project that provides coordination services for distributed frameworks.

1.2 Zookeeper working mechanism

Zookeeper is understood from the perspective of design patterns: it is a distributed service management framework designed based on the observer pattern. It is responsible for storing and managing the data that everyone cares about, and then accepts the registration of observers. Once the state of these data changes, Zookeeper will will be responsible for notifying those observers registered with Zookeeper to react accordingly. That is to say Zookeeper = file system + notification mechanism.

1.3 Zookeeper features

1. Zookeeper: a leader (Leader), a cluster of multiple followers (Follower).
2. As long as more than half of the nodes in the Zookeeper cluster survive, the Zookeeper cluster can serve normally. So Zookeeper is suitable for installing an odd number of servers.
3. Global data consistency: Each server saves a copy of the same data, and the data is consistent no matter which server the client connects to.
4. The update requests are executed sequentially, and the update requests from the same Client are executed sequentially in the order in which they are sent, that is, first in first out.
5. Data update atomicity, a data update either succeeds or fails.
6. Real-time, within a certain time range, the Client can read the latest data.

1.4 Zookeeper data structure

The structure of the ZooKeeper data model is very similar to the Linux file system, which can be regarded as a tree as a whole, and each node is called a ZNode. Each ZNode can store 1MB of data by default, and each ZNode can be uniquely identified by its path.

1.5 Zookeeper application scenarios

The services provided include: unified naming service, unified configuration management, unified cluster management, dynamic online and offline server nodes, soft load balancing, etc.

• Unified Naming Service
  In a distributed environment, it is often necessary to uniformly name applications/services for easy identification. For example: IP is not easy to remember, but domain name is easy to remember.

• Unified configuration management
  (1) In a distributed environment, configuration file synchronization is very common. It is generally required that in a cluster, the configuration information of all nodes is consistent, such as Kafka cluster. After the configuration file is modified, it is hoped that it can be quickly synchronized to each node.
  (2) Configuration management can be implemented by ZooKeeper. Configuration information can be written to a Znode on ZooKeeper. Each client server listens to this Znode. Once the data in Znode is modified, ZooKeeper will notify each client server.

• Unified cluster management
  (1) In a distributed environment, it is necessary to know the status of each node in real time. Some adjustments can be made according to the real-time status of the nodes.
  (2) ZooKeeper can realize real-time monitoring of node status changes. Node information can be written to a ZNode on ZooKeeper. Listening to this ZNode can obtain its real-time status changes.

• The server dynamically goes online and offline
  The client can see the changes of the server going online and offline in real time.

• Soft load balancing
  records the number of visits of each server in Zookeeper, and lets the server with the least number of visits handle the latest client requests.

1.6 Zookeeper election mechanism

• Start the election mechanism for the first time
  (1) Server 1 starts and initiates an election. Server 1 votes for itself. At this time, server 1 has one vote, less than half (3 votes), the election cannot be completed, and the status of server 1 remains LOOKING;
  (2) Server 2 is started, and another election is initiated. Servers 1 and 2 cast their own votes and exchange ballot information: At this time, server 1 finds that the myid of server 2 is larger than the one currently voted for (server 1), and changes the vote to recommend server 2. At this time, server 1 has 0 votes and server 2 has 2 votes. If there is no more than half of the results, the election cannot be completed, and the status of servers 1 and 2 remains LOOKING.
  (3) Server 3 starts and initiates an election. At this point, both servers 1 and 2 will change their votes to server 3. The result of this vote: Server 1 has 0 votes, Server 2 has 0 votes, and Server 3 has 3 votes. At this time, server 3 has more than half of the votes, and server 3 is elected as the leader. Servers 1 and 2 change their status to FOLLOWING, and server 3 changes their status to LEADING;
  (4) Server 4 starts and initiates an election. At this time, servers 1, 2, and 3 are no longer in the LOOKING state, and the ballot information will not be changed. The result of exchanging ballot information: Server 3 has 3 votes, and Server 4 has 1 vote. At this time, server 4 obeys the majority, changes the ballot information to server 3, and changes the status to FOLLOWING; (5) server 5 starts, and acts as a younger brother like 4.

• The election mechanism is not started for the first time
  (1) When a server in the ZooKeeper cluster has one of the following two situations, it will start to enter the Leader election:
  1) The server is initialized and started.
  2) The connection to the Leader cannot be maintained while the server is running.

(2) When a machine enters the Leader election process, the current cluster may also be in the following two states:
1) There is already a Leader in the cluster.
For the case where a leader already exists, when the machine tries to elect a leader, it will be informed of the leader information of the current server. For this machine, it only needs to establish a connection with the leader machine and perform state synchronization.

2) There is indeed no Leader in the cluster.
Suppose ZooKeeper consists of 5 servers, the SIDs are 1, 2, 3, 4, and 5, and the ZXIDs are 8, 8, 8, 7, and 7, and the server with SID 3 is the leader. At some point, servers 3 and 5 fail, so a Leader election begins.
Election Leader rules:
1. The one with the larger EPOCH wins directly
2. The one with the same EPOCH and the larger transaction id wins
3. The one with the same transaction id and the larger server id wins

SID: server ID. It is used to uniquely identify a machine in a ZooKeeper cluster. Each machine cannot be repeated, and it is consistent with myid.
ZXID: Transaction ID. ZXID is a transaction ID used to identify a server state change. At a certain moment, the ZXID value of each machine in the cluster may not be exactly the same, which is related to the processing logic speed of the ZooKeeper server for the client "update request".
Epoch: The code name for each Leader term. The logical clock value in the same round of voting process is the same when there is no Leader. This number increases each time a vote is cast

2. Deploy the Zookeeper cluster

2.1 Environment deployment

//准备 3 台服务器做 Zookeeper 集群
192.168.80.10
192.168.80.11
192.168.80.12

2.2 Turn off the firewall

1.安装前准备
//关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
setenforce 0

2.3 Install JDK Download the installation package and install Zookeeper

//安装 JDK
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
java -version

//下载安装包
官方下载地址：https://archive.apache.org/dist/zookeeper/

cd /opt
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.5.7/apache-zookeeper-3.5.7-bin.tar.gz


2.安装 Zookeeper
cd /opt
tar -zxvf apache-zookeeper-3.5.7-bin.tar.gz
mv apache-zookeeper-3.5.7-bin /usr/local/zookeeper-3.5.7

2.4 Modify the configuration file

cd /usr/local/zookeeper-3.5.7/conf/
cp zoo_sample.cfg zoo.cfg

vim zoo.cfg
tickTime=2000   #通信心跳时间，Zookeeper服务器与客户端心跳时间，单位毫秒
initLimit=10    #Leader和Follower初始连接时能容忍的最多心跳数（tickTime的数量），这里表示为10*2s
syncLimit=5     #Leader和Follower之间同步通信的超时时间，这里表示如果超过5*2s，Leader认为Follwer死掉，并从服务器列表中删除Follwer
dataDir=/usr/local/zookeeper-3.5.7/data      ●修改，指定保存Zookeeper中的数据的目录，目录需要单独创建
dataLogDir=/usr/local/zookeeper-3.5.7/logs   ●添加，指定存放日志的目录，目录需要单独创建
clientPort=2181   #客户端连接端口
#添加集群信息
server.1=192.168.80.10:3188:3288
server.2=192.168.80.11:3188:3288
server.3=192.168.80.12:3188:3288

server.A=B:C:D
●A is a number, indicating which server number this is. In cluster mode, you need to create a file myid in the directory specified by dataDir in zoo.cfg. There is a data in this file that is the value of A. When Zookeeper starts, read this file, get the data in it and the configuration in zoo.cfg Compare the information to determine which server it is.
● B is the address of this server.
●C is the port through which the Follower of this server exchanges information with the Leader server in the cluster.
●D means that in case the Leader server in the cluster hangs up, a port is needed to re-elect to elect a new Leader, and this port is the port used for communication between the servers during the election.

2.5 Copy the configuration to other servers

//拷贝配置好的 Zookeeper 配置文件到其他机器上
scp /usr/local/zookeeper-3.5.7/conf/zoo.cfg 192.168.80.11:/usr/local/zookeeper-3.5.7/conf/
scp /usr/local/zookeeper-3.5.7/conf/zoo.cfg 192.168.80.12:/usr/local/zookeeper-3.5.7/conf/

//在每个节点上创建数据目录和日志目录
mkdir /usr/local/zookeeper-3.5.7/data
mkdir /usr/local/zookeeper-3.5.7/logs

//在每个节点的dataDir指定的目录下创建一个 myid 的文件
echo 1 > /usr/local/zookeeper-3.5.7/data/myid
echo 2 > /usr/local/zookeeper-3.5.7/data/myid
echo 3 > /usr/local/zookeeper-3.5.7/data/myid

2.6 Configure Zookeeper startup script

//配置 Zookeeper 启动脚本
vim /etc/init.d/zookeeper
#!/bin/bash
#chkconfig: 2345 20 90
#description:Zookeeper Service Control Script
ZK_HOME='/usr/local/zookeeper-3.5.7'
case $1 in
start)
	echo "---------- zookeeper 启动 ------------"
	$ZK_HOME/bin/zkServer.sh start
;;
stop)
	echo "---------- zookeeper 停止 ------------"
	$ZK_HOME/bin/zkServer.sh stop
;;
restart)
	echo "---------- zookeeper 重启 ------------"
	$ZK_HOME/bin/zkServer.sh restart
;;
status)
	echo "---------- zookeeper 状态 ------------"
	$ZK_HOME/bin/zkServer.sh status
;;
*)
    echo "Usage: $0 {start|stop|restart|status}"
esac

//	设置开机自启
chmod +x /etc/init.d/zookeeper
chkconfig --add zookeeper

//分别启动 Zookeeper
service zookeeper start

//查看当前状态
service zookeeper status

3. Overview of Kafka

3.1 Why message queue - (MQ)

• The main reason is that in a high-concurrency environment, synchronous requests are too late to process, and requests often block. For example, a large number of requests access the database concurrently, resulting in row locks and table locks. In the end, too many request threads will accumulate, which will trigger too many connection errors and cause an avalanche effect.
• We use message queues to ease the pressure on the system by processing requests asynchronously. Message queues are often used in scenarios such as asynchronous processing, traffic peak shaving, application decoupling, and message communication.

Currently, the more common MQ middleware include ActiveMQ, RabbitMQ, RocketMQ, Kafka, etc.

3.2 Benefits of using message queues

(1) Decoupling
allows you to independently expand or modify the processing on both sides, as long as they abide by the same interface constraints.

(2) Recoverability
When a part of the system fails, it will not affect the whole system. The message queue reduces the coupling between processes, so even if a process that processes messages hangs up, the messages added to the queue can still be processed after the system recovers.

(3) Buffering
helps to control and optimize the speed of data flow through the system, and solve the inconsistency in the processing speed of production messages and consumption messages.

(4) Flexibility & peak processing capacity
In the case of a sharp increase in traffic, the application still needs to continue to function, but such burst traffic is not common. It is undoubtedly a huge waste to invest resources on standby at all times to handle such peak access. The use of message queues can enable key components to withstand sudden access pressure without completely crashing due to sudden overload requests.

(5) Asynchronous communication
In many cases, users do not want or need to process messages immediately. Message queues provide an asynchronous processing mechanism that allows users to put a message into a queue without processing it immediately. Put as many messages on the queue as you want, and process them when needed.

3.3 Two modes of message queue

(1) Point-to-point mode (one-to-one, the consumer actively pulls the data, and the message is cleared after the message is received)
The message producer produces a message and sends it to the message queue, and then the message consumer takes it out from the message queue and consumes the message. After the message is consumed, there is no more storage in the message queue, so it is impossible for the message consumer to consume the message that has already been consumed. The message queue supports multiple consumers, but for a message, only one consumer can consume it.

(2) Publish/subscribe mode (one-to-many, also known as observer mode, the message will not be cleared after the consumer consumes the data) the
message producer (publish) publishes the message to the topic, and there are multiple message consumers (subscribe ) to consume the message. Unlike the peer-to-peer method, messages published to a topic will be consumed by all subscribers.
The publish/subscribe mode defines a one-to-many dependency relationship between objects, so that whenever the state of an object (target object) changes, all objects (observer objects) that depend on it will be notified and automatically updated.

3.4 kafka definition

Kafka is a distributed publish/subscribe-based message queue (MQ, Message Queue), which is mainly used in real-time computing and log collection in the field of big data.

3.5 kafka introduction

Kafka was originally developed by Linkedin. It is a distributed, partition-supporting, replica-based distributed message middleware system based on Zookeeper coordination. Its biggest feature is that it can process large amounts of data in real time. To meet various demand scenarios, such as hadoop-based batch processing system, low-latency real-time system, Spark/Flink streaming processing engine, nginx access log, message service, etc., written in scala language, Linkedin contributed to it in
2010 The Apache Foundation and become a top open source project.

3.6 Features of Kafka

• High-throughput, low-latency
  Kafka can process hundreds of thousands of messages per second, and its latency is as low as a few milliseconds. Each topic can be divided into multiple Partitions, and the Consumer Group performs consumption operations on the Partitions to improve load balancing and consumption capabilities.

• Scalability
  Kafka cluster supports hot expansion

• Persistence and reliability
  Messages are persisted to local disks, and data backup is supported to prevent data loss

• Fault tolerance
  allows nodes in the cluster to fail (in the case of multiple copies, if the number of copies is n, n-1 nodes are allowed to fail)

• High concurrency
  Support thousands of clients to read and write at the same time

3.7 kafka system architecture

(1) Broker
A kafka server is a broker. A cluster consists of multiple brokers. A broker can accommodate multiple topics.

(2) Topic
can be understood as a queue, and both producers and consumers face the same topic.
Similar to the table name of the database or the index of the ES,
the messages of different topics are stored separately

(3) Partition
In order to achieve scalability, a very large topic can be distributed to multiple brokers (ie servers), a topic can be divided into one or more partitions, and each partition is an ordered queue. Kafka only guarantees that the records in the partition are in order, but does not guarantee the order of different partitions in the topic.

Each topic has at least one partition. When the producer generates data, it will select a partition according to the allocation strategy, and then append the message to the end of the queue of the specified partition.
##Partation Data routing rules:
1．If patition is specified, use it directly;
2. 3. If no patition is specified but a key is specified (equivalent to an attribute in the message), a patition is selected by performing hash modulo on the value of the key;
3．Both patition and key are not specified, and a patition is selected by polling.

Each message will have a self-incrementing number, which is used to identify the offset of the message, and the identification sequence starts from 0.

Data in each partition is stored using multiple segment files.

If the topic has multiple partitions, the order of the data cannot be guaranteed when consuming data. In the scenario where the order of consumption of messages is strictly guaranteed (for example, flash sales of products and grabbing red envelopes), the number of partitions needs to be set to 1.

• Broker stores topic data. If a topic has N partitions and the cluster has N brokers, each broker stores a partition of the topic.
• If a topic has N partitions and the cluster has (N+M) brokers, then there are N brokers that store a partition of the topic, and the remaining M brokers do not store the partition data of the topic.
• If a topic has N partitions and the number of brokers in the cluster is less than N, then one broker stores one or more partitions of the topic. In the actual production environment, try to avoid this situation, which can easily lead to data imbalance in the Kafka cluster.

3.8 Reasons for Partitioning

• It is convenient to expand in the cluster. Each Partition can be adjusted to adapt to the machine where it is located, and a topic can be composed of multiple Partitions, so the entire cluster can adapt to data of any size;
• Concurrency can be improved because it can be read and written in units of Partition.

(4) Replica
copy. In order to ensure that when a node in the cluster fails, the partition data on the node will not be lost, and Kafka can still continue to work. Kafka provides a copy mechanism. Each partition of a topic has several Replica, a leader and several followers.

(5)
Each partition of the leader has multiple copies, and only one of them is the leader. The leader is currently responsible for reading and writing data.

(6) Follower
Follower follows the Leader, all write requests are routed through the Leader, data changes will be broadcast to all Followers, and the Follower and Leader maintain data synchronization. Follower is only responsible for backup, not for reading and writing data.
If the Leader fails, a new Leader is elected from the Followers.
When the Follower hangs, gets stuck, or is too slow to synchronize, the Leader will delete the Follower from the ISR (a set of Followers maintained by the Leader that is synchronized with the Leader) list, and create a new Follower.

(7) Producer
The producer is the publisher of the data. This role publishes the message push to the topic of Kafka.
After the broker receives the message sent by the producer, the broker appends the message to the segment file currently used for appending data.
The message sent by the producer is stored in a partition, and the producer can also specify the partition of the data storage.

(8) Consumers
can pull data from the broker. Consumers can consume data from multiple topics.

(9) Consumer Group (CG)
The consumer group consists of multiple consumers.
All consumers belong to a consumer group, that is, a consumer group is a logical subscriber. A group name can be specified for each consumer, and if no group name is specified, it belongs to the default group.
Collecting multiple consumers together to process the data of a certain topic can improve the consumption capacity of data faster.
Each consumer in the consumer group is responsible for consuming data from different partitions. A partition can only be consumed by one consumer in the group to prevent data from being read repeatedly.
Consumer groups do not affect each other.

(10) offset
can uniquely identify a message.
The offset determines the location of the read data, and there will be no thread safety issues. The consumer uses the offset to determine the message to be read next time (that is, the consumption location).
After the message is consumed, it is not deleted immediately, so that multiple businesses can reuse Kafka messages.
A certain service can also achieve the purpose of re-reading messages by modifying the offset, which is controlled by the user.
The message will eventually be deleted, and the default life cycle is 1 week (7*24 hours).

(11) Zookeeper
Kafka uses Zookeeper to store the meta information of the cluster.

Since the consumer may experience failures such as power outages and downtime during the consumption process, after the consumer recovers, it needs to continue to consume from the location before the failure. Therefore, the consumer needs to record which offset it consumes in real time, so that it can continue to consume after the failure recovers.
Before Kafka version 0.9, the consumer saved the offset in Zookeeper by default; starting from version 0.9, the consumer saved the offset in a built-in Kafka topic by default, which is __consumer_offsets.

That is to say, the role of zookeeper is that when producers push data to the kafka cluster, they must find out where the nodes of the kafka cluster are, and these are all found through zookeeper. Which piece of data the consumer consumes also needs the support of zookeeper. The offset is obtained from zookeeper, and the offset records where the last consumed data was consumed, so that the next piece of data can be consumed next.

Summarize

Zookeeper   分布式系统管理框架，主要用来解决分布式集群中应用系统的一致性问题    相当于各种分布式应用服务的 注册中心 + 文件系统 + 通知机制
本质： 用于注册各种各种分布式应用，存储和管理各种分布式应用服务的元数据，如果应用服务状态发生改变会通知客户端

Zookeeper 选举机制
第一次选举：比较服务器节点的myid，谁myid大就获取比它小的服务器节点的选票，当选票超过节点服务器数量的半数则当选为leader，其它节点为follower，即使后面再有其它myid更大的节点加入到集群也不会影响之前的选举结果。

非第一次选举：如果是非leader节点故障，替换的新节点继续当follower，与leader对接并同步数据
              如果是leader节点故障，则需要重新选举新leader，先比较每个节点的Epoch（参加选举的次数），选最大的当leader
			                                                若Epoch有相同的节点，则再比较ZXID（写操作的事务ID），选ZXID最大的当leader
															若ZXID也有相同的节点，则再比较SID（等同于myid），选SID最大的当leader
															
															
中间件   消息队列型（MQ）  ActiveMQ、RabbitMQ、RocketMQ、Kafka、Pulsar、Redis
         Web应用型（代理服务器）  Nginx、Haproxy、LVS、Tomcat、php
	
	
消息队列的作用：应用解耦、异步处理、流量削峰、缓冲
		 
消息队列的模式：
点对点模式 ：一对一，消费者消费消息后会删除消息
发布/订阅模式（观察者模式）：一对多，消费者消费后不会删除消息
		 
Kafka架构
broker    kafka服务器节点
producer  生产者，发布消息到topic
consumer  消费者    
consumer group   消费者组，是消息的实际订阅者，一个消费者组包含一个或多个消费者（组内成员不能重复消费同一个partition数据）  

producer -> topic消息队列 -> partition分区 -> replica副本（leader负责数据读写、follower只负责同步复制leader的数据）

consumer -> offset偏移量（用来记录消费者上一次消费的位置） 

zookeeper 存储kafka集群的元数据信息，生产者和消费者的动作都需要zookeeper的管理和支持。
          比如生产者推送数据到kafka集群需要通过zookeeper去寻找kafka服务器节点的位置，消费者需要从zookeeper获取offset记录的上一次消费的位置继续往后消费
		  
kafka 只能保证 partition分区内的消息顺序，消费时无法保证 partition 之间的顺序。
如需要严格保证消息的消费顺序（商品秒杀、抢红包等场景）要把 partition 数据设置为 1