An article with you to understand architecture ZooKeeper

Previous article, we explained the ZooKeeper to get started, this article focuses on the next ZooKeeper architecture, understanding ZooKeeper architecture can help us better design collaboration services.

First we look at the overall architecture diagram of ZooKeeper.

ZooKeeper overall architecture

Applications using ZooKeeper ZooKeeper client library to use the service, the establishment of a session of a client and ZooKeeper node in the cluster, ZooKeeper client is responsible for interaction and ZooKeeper cluster. ZooKeeper cluster can have two modes: standalone mode and quorum mode. ZooKeeper ZooKeeper node cluster in standalone mode as well as a stand-alone, standalone mode is typically used to develop. Usually in production environments ZooKeeper quorum mode, the plurality of replacement ZooKeeper ZooKeeper nodes in the cluster quorum mode.

Session

Session is an important concept ZooKeeper client, create a session ZooKeeper ZooKeeper client library and nodes in the cluster. The client can take the initiative to close the session. Also, if the node does not receive data ZooKeeper client within the timeout time associated with the session, then, the node will ZooKeeper closed session. In addition ZooKeeper ZooKeeper client library If you find a connection error, it will automatically establish a connection with other ZooKeeper nodes.

The figure below shows how ZooKeeper client is re-connected?

At first ZooKeeper ZooKeeper session client and cluster node 1 established after a period of time, ZooKeeper node 1 fails, the client automatically and ZooKeeper ZooKeeper cluster node 3 to re-establish a session connection.

Quorum mode

处于 Quorum 模式的 ZooKeeper 集群包含多个 ZooKeeper 节点。 下图的 ZooKeeper 集群有 3 个节点，其中节点 1 是 leader 节点，节点 2 和节点 3 是 follower 节点。集群中只能有一个 leader 节点，可以有多个 follower 节点，leader 节点可以处理读写请求，follower 只可以处理读请求。follower 在接到写请求时会把写请求转发给 leader 来处理。

下面来说下 ZooKeeper 保证的数据一致性：

数据一致性

• 可线性化（Linearizable）写入：先到达 leader 的写请求会被先处理，leader 决定写请求的执行顺序。
• 客户端 FIFO 顺序：来自给定客户端的请求按照发送顺序执行。

为了让大家更好地理解 Quorum 模式，下面会配置一个 3 节点的 Quorum 模式的 ZooKeeper 集群。

搭建 3 节点 ZooKeeper 集群

首先需要准备 3 个配置文件，dataDir 和 clientPort 配置项要配置不同的值。3 个配置文件的 server.n 部分都是一样的。在 server.1=127.0.0.1:3333:3334，其中 3333 是 quorum 之间的通信的端口号，3334 是用于 leader 选举的端口号。

还需要在每个节点的 dataDir 目录下创建 myid 文件，里面内容为一个数字，用来标识当前主机，配置文件中配置的 server.n 中 n 为什么数字，则 myid 文件中就输入这个数字。

如下是第 1 个节点的配置文件，其中目录是 node1，端口号用的是 2181，另外两个节点的目录分别是 node2 和 node3，端口号分别为 2182 和 2183，最下面的三行都是一样的：

# 心跳检查的时间 2秒
tickTime=2000
# 初始化时 连接到服务器端的间隔次数，总时间10*2=20秒
initLimit=10
# ZK Leader 和follower 之间通讯的次数，总时间5*2=10秒
syncLimit=5
# 存储内存中数据库快照的位置，如果不设置参数，更新事务日志将被存储到默认位置。
dataDir=/data/zk/quorum/node1
# ZK 服务器端的监听端口  
clientPort=2181

server.1=127.0.0.1:3333:3334
server.2=127.0.0.1:4444:4445
server.3=127.0.0.1:5555:5556

下面来启动这个集群，首先启动第一个节点，启动命令如下：

zkServer.sh start-foreground /usr/local/apache-zookeeper-3.5.6-bin/conf/zoo-quorum-node1.cfg 

Note: start-foreground option zkServer.sh running in the foreground, the log directly hit the console. If we hit the log file, the log will hit three zkServer.sh the same file.

In the first start after the log node appears as follows:

2019-12-29 13:14:35,758 [myid:1] - WARN  [WorkerSender[myid=1]:QuorumCnxManager@679] - Cannot open channel to 2 at election address /127.0.0.1:4445
java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707)
    at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620)
    at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477)
    at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456)
    at java.lang.Thread.run(Thread.java:748)

The reason for the three node configuration file configuration, but only started one node, at present he and the other two other nodes can not establish a connection, it is reported this problem.

Then start the second node, run the following command:

zkServer.sh start-foreground /usr/local/apache-zookeeper-3.5.6-bin/conf/zoo-quorum-node2.cfg 

After the start, we can find in the log node 2 in such a line:

2019-12-29 13:15:13,699 [myid:2] - INFO  [QuorumPeer[myid=2](plain=/0.0.0.0:2182)(secure=disabled):Leader@464] - LEADING - LEADER ELECTION TOOK - 41 MS

This indicates that the node 2 becomes the leader node, can also be found in the following line of the log in the log node 1, node 1 became described follower node.

2019-12-29 13:15:13,713 [myid:1] - INFO  [QuorumPeer[myid=1](plain=/0.0.0.0:2181)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 61 MS

For a three-node cluster because it represents the most two, Quorum pattern is formed, the next start of the third node, run the following command:

zkServer.sh start-foreground /usr/local/apache-zookeeper-3.5.6-bin/conf/zoo-quorum-node3.cfg

After starting in the log has the following line, a third node joined the cluster, and a node is added as a follower.

2019-12-29 13:15:52,440 [myid:3] - INFO  [QuorumPeer[myid=3](plain=/0.0.0.0:2183)(secure=disabled):Follower@69] - FOLLOWING - LEADER ELECTION TOOK - 15 MS

Let's start the client to use this three-node cluster, add in the command -serveroption specified later is the host name and port number three nodes, the command is as follows:

zkCli.sh -server 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183

2019-12-29 13:45:44,982 [myid:127.0.0.1:2181] - INFO  [main-SendThread(127.0.0.1:2181):ClientCnxn$SendThread@1394] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x101fff740830000, negotiated timeout = 30000

Logs can be seen by starting the client and port number 2181 to establish a connection node, which is the first node and a connection is established.

We execute ls -R /commands look znode data in this cluster.

[zk: 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183(CONNECTED) 1] ls -R /
/
/zookeeper
/zookeeper/config
/zookeeper/quota

Let's kill a ZooKeeper node, to see whether the client can re-connect. Now we even node 1, the node 1 kill us, you can find the client and the port number 2183 of the node to re-establish a connection, that is, and established a connection node 3 in the client's log.

2019-12-29 14:03:31,392 [myid:127.0.0.1:2183] - INFO  [main-SendThread(127.0.0.1:2183):ClientCnxn$SendThread@1394] - Session establishment complete on server localhost/127.0.0.1:2183, sessionid = 0x101fff740830000, negotiated timeout = 30000

Then we look at the client are working correctly, execution ls -R /, data may be found to be able to return to normal, indicating that the client is able to normal use.

[zk: 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183(CONNECTED) 4] ls -R /
/
/zookeeper
/zookeeper/config
/zookeeper/quota

to sum up

This article explains the ZooKeeper architecture, and how to configure a three-node cluster ZooKeeper.