Recently due to the high availability of business requirements, testing various components. Because of our environment deployed in the AWS Beijing. Only two Aviable Zone (usable area).
Notes: There are two data centers, each needs to do disaster recovery, and paper test case is the same.
The Zookeeper need more than three singular nodes simultaneously, and must ensure that more than half of the nodes survive, but also to provide normal service.
So, for the case of only two AZ, regardless of how planning, AZ has a probability of experiencing the presence of more than half of the hang, resulting in the entire Zookeeper unavailable.
So we can do is hang in AZ after this, how do we deal with as soon as possible and restore the environment.
We are prepared two software installed, configured machine parameters. After a complete hang usable region can be activated manually two spare nodes. The number of available Zookeeper region 2 is increased majority. 2 Zookeeper can restore service in the available area.
Referring next to FIG:
Above assumptions, if you can achieve it?
Then we test it today.
1. prepared a total of five machines, as a test
Download and install 2. Zookeeper's.
2.1 Zookeeper official Download
https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/
2.2 software download
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
2.3 Zookeeper detailed installation steps, refer to:
https://blog.51cto.com/hsbxxl/1971241
2.4 zoo.cfg configuration #cat zoo.cfg
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.1=172.31.9.73:2888:3888 server.2=172.31.20.233:2888:3888 server.3=172.31.26.111:2888:3888 server.4=172.31.17.68:2888:3888 server.5=172.31.16.33:2888:3888
2.5 根据zoo.cfg创建data和log两个文件夹
mkdir -p /data/zookeeper/data mkdir -p /data/zookeeper/log
2.6 根据节点号码，修改文件
echo 1 > /data/zookeeper/data/myid
3. 一共准备了5台EC2进行测试，并且都已经安装好Zookeeper
但是只启动三台，另两个机器作为standby
下图可以看到，已经有三台启动zookeeper，
注意，在Zookeeper启动的过程中，必须保证三台及以上，zookeeper集群才能正常工作
4. 接下来，我开始逐个机器关机，看zookeeper的状态
当前leader在zk3上，我们先关闭zk1，再关闭zk3，看Leader会不会飘到zk2上
4.1 在zk1上执行kill，杀掉进程
[root@ip-172-31-9-73 ~]# jps 12438 Jps 7545 QuorumPeerMain [root@ip-172-31-9-73 ~]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /root/zookeeper-3.4.14/bin/../conf/zoo.cfg Mode: follower [root@ip-172-31-9-73 ~]# kill -9 7545
4.2 在zk5上通过zkCli链接zk3，并可以查询数据。
在zk1上kill掉进程之后，理论上，还有zk2和zk3存活，但是zkCli的连接显示已经报错。
[root@ip-172-31-16-33 bin]# ./zkCli.sh -server 172.31.26.111:2181 Connecting to 172.31.26.111:2181 ...... [zk: 172.31.26.111:2181(CONNECTED) 0] ls / [zk-permanent, zookeeper, test] [zk: 172.31.26.111:2181(CONNECTED) 1] 2019-06-23 07:28:06,581 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x30000c504530000, likely server has closed socket, closing socket connection and attempting reconnect ...... 2019-06-23 07:28:09,822 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-23 07:28:09,824 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@879] - Socket connection established to ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181, initiating session 2019-06-23 07:28:09,825 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1158] - Unable to read additional data from server sessionid 0x30000c504530000, likely server has closed socket, closing socket connection and attempting reconnect
4.3 我们继续 kill掉zk3上的进程，只保留zk2上的进程。但是我们已经无法确认zk2是Leader还是Follow，或者说，他是否还保留有数据。
[root@ip-172-31-26-111 bin]# jps 4183 QuorumPeerMain 4648 Jps [root@ip-172-31-26-111 bin]# kill -9 4183 [root@ip-172-31-26-111 bin]# jps 4658 Jps
4.4 zk3上进程kill掉之后，链接就不只是上面的报错了，而是直接连接拒绝
[root@ip-172-31-16-33 bin]# ./zkCli.sh -server 172.31.26.111:2181 Connecting to 172.31.26.111:2181 ...... Welcome to ZooKeeper! 2019-06-23 07:35:18,411 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181. Will not attempt to authenticate using SASL (unknown error) JLine support is enabled 2019-06-23 07:35:18,533 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1162] - Socket error occurred: ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181: Connection refused [zk: 172.31.26.111:2181(CONNECTING) 0] 2019-06-23 07:35:19,639 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181. Will not attempt to authenticate using SASL (unknown error) 2019-06-23 07:35:19,640 [myid:] - INFO [main-SendThread(ip-172-31-26-111.cn-north-1.compute.internal:2181):ClientCnxn$SendThread@1162] - Socket error occurred: ip-172-31-26-111.cn-north-1.compute.internal/172.31.26.111:2181: Connection refused
4.5 可以看到zk2上的进程还在，
# jps 5155 QuorumPeerMain 5211 Jps
4.6 并且通过下面命令，可以检查到zk2 的2181端口还在提供服务
# echo ruok | nc localhost 2181 imok
4.7 但是其他命令是没有正常输出的，只有echo ruok | nc localhost 2181输出ok。
# echo ruok | nc 172.31.16.33 2181 imok[root@ip-172-31-16-33 bin]# echo conf | nc 172.31.16.33 2181 This ZooKeeper instance is not currently serving requests # echo dump | nc 172.31.16.33 2181 This ZooKeeper instance is not currently serving requests
4.8 ZooKeeper 四字命令
ZooKeeper 四字命令 |
功能描述 |
conf |
输出相关服务配置的详细信息。 |
cons |
列出所有连接到服务器的客户端的完全的连接 / 会话的详细信息。包括“接受 / 发送”的包数量、会话 id 、操作延迟、最后的操作执行等等信息。 |
dump |
列出未经处理的会话和临时节点。 |
envi |
输出关于服务环境的详细信息（区别于 conf 命令）。 |
reqs |
列出未经处理的请求 |
ruok |
测试服务是否处于正确状态。如果确实如此，那么服务返回“imok ”，否则不做任何相应。 |
stat |
输出关于性能和连接的客户端的列表。 |
wchs |
列出服务器 watch 的详细信息。 |
wchc |
通过 session 列出服务器 watch 的详细信息，它的输出是一个与watch 相关的会话的列表。 |
wchp |
4.9 正常情况下，以上命令可以输出：
# echo dump | nc 172.31.20.233 2181
SessionTracker dump: org.apache.zookeeper.server.quorum.LearnerSessionTracker@77714302 ephemeral nodes dump: Sessions with Ephemerals (0):
# echo conf | nc 172.31.20.233 2181
clientPort=2181 dataDir=/data/zookeeper/data/version-2 dataLogDir=/data/zookeeper/log/version-2 tickTime=2000 maxClientCnxns=60 minSessionTimeout=4000 maxSessionTimeout=40000 serverId=2 initLimit=10 syncLimit=5 electionAlg=3 electionPort=3888 quorumPort=2888 peerType=0
# echo envi| nc 172.31.20.233 2181
Environment: zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT host.name=ip-172-31-20-233.cn-north-1.compute.internal java.version=1.8.0_212 java.vendor=Oracle Corporation java.home=/usr/java/jdk1.8.0_212-amd64/jre java.class.path=/root/zookeeper-3.4.14/bin/../zookeeper-server/target/classes:/root/zookeeper-3.4.14/bin/../build/classes:/root/zookeeper-3.4.14/bin/../zookeeper-server/target/lib/*.jar:/root/zookeeper-3.4.14/bin/../build/lib/*.jar:/root/zookeeper-3.4.14/bin/../lib/slf4j-log4j12-1.7.25.jar:/root/zookeeper-3.4.14/bin/../lib/slf4j-api-1.7.25.jar:/root/zookeeper-3.4.14/bin/../lib/netty-3.10.6.Final.jar:/root/zookeeper-3.4.14/bin/../lib/log4j-1.2.17.jar:/root/zookeeper-3.4.14/bin/../lib/jline-0.9.94.jar:/root/zookeeper-3.4.14/bin/../lib/audience-annotations-0.5.0.jar:/root/zookeeper-3.4.14/bin/../zookeeper-3.4.14.jar:/root/zookeeper-3.4.14/bin/../zookeeper-server/src/main/resources/lib/*.jar:/root/zookeeper-3.4.14/bin/../conf: java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib java.io.tmpdir=/tmp java.compiler=<NA> os.name=Linux os.arch=amd64 os.version=4.14.123-86.109.amzn1.x86_64 user.name=root user.home=/root user.dir=/root/zookeeper-3.4.14/bin
5. 这个时候，我去启动另外两个备用节点，zk4，zk5.这个两个节点都是第一次启动。
6. 再次连接到zookeeper上，可以看到，至少数据还是没有丢失的
[root@ip-172-31-16-33 bin]# ./zkCli.sh -server 172.31.16.33:2181 Connecting to 172.31.16.33:2181 ...... [zk: 172.31.16.33:2181(CONNECTED) 0] ls / [zk-permanent, zookeeper, test]
7. 通过以上测试，似乎是达到我们预期的结果。唯一的一点小问题，就是：我们有3个节点，为什么关闭1个，剩余两个，就不能正常运行了呢？
其实，这里是有个“想当然”的小问题。
我们以为，只启动三个. 其实，Zookeeper集群，识别的是5个, 为什么呢？
Zookeeper靠什么去识别集群中有几个节点呢？当然不是靠“想当然”。一定是有配置文件告诉它。Zookeeper，只有两个配置文件zoo.cfg和myid。
那就只有zoo.cfg会影响到它了。
8. 我将zoo.cfg做如下修改之后。只开启3个节点，在关闭一个节点之后，还是可以正常运行的。
注释掉server2和server5
# cat zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.1=172.31.9.73:2888:3888 #server.2=172.31.20.233:2888:3888 server.3=172.31.26.111:2888:3888 server.4=172.31.17.68:2888:3888 #server.5=172.31.16.33:2888:3888
9. 关闭server4之后，还有server2和server3活着。
[root@ip-172-31-26-111 ~]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /root/zookeeper-3.4.14/bin/../conf/zoo.cfg Mode: leader [root@ip-172-31-9-73 ~]# zkServer.sh status ZooKeeper JMX enabled by default Using config: /root/zookeeper-3.4.14/bin/../conf/zoo.cfg Mode: follower
10. 总结，如果考虑两个AZ的情况下，zookeeper节点数多的AZ出现灾难情况，我们如何快速恢复？
(假设Server1/Server2在1AZ，Server3/Server4/Server5在2AZ)
10.1. 在Zookeeper节点少的AZ，多准备2台配置好zookeeper的EC2，并关机待使用。Server4/Server5具体zoo.cfg配置如下
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.3=172.31.26.111:2888:3888 server.4=172.31.17.68:2888:3888 server.5=172.31.16.33:2888:3888
10.2. Server1/Server2/Server3，是正常运行的节点，配置如下：
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.1=172.31.9.73:2888:3888 server.2=172.31.20.233:2888:3888 server.3=172.31.26.111:2888:3888
10.3. 灾难发生，Server1/Server2所在的1AZ挂掉的情况下，需要人工介入，将Server3的配置更改为如下配置，并重启Server3的zookeeper服务，然后启动Server4/Server5，一定要先启动Server3，注意顺序。
tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=6 server.3=172.31.26.111:2888:3888 server.4=172.31.17.68:2888:3888 server.5=172.31.16.33:2888:3888
10.4 日常运行状态
10.5 检查已经创建的znode信息
./zkCli.sh -server 172.31.16.33:2181 ls / Connecting to 172.31.16.33:2181 [zk-permanent, zookeeper, test]
10.6 关闭Server1/Server2，注意顺序，先关闭follow，如果先关闭leader，会发生切换。我们期望的是Server3最后以follow的身份存活。
11. 最终可以看到测试结果，一切都是按照我们“想当然”的方向发展。
12. 最后验证zookeeper中的znode数据,还是都存在的。
./zkCli.sh -server 172.31.16.33:2181 ls / Connecting to 172.31.16.33:2181 [zk-permanent, zookeeper, test]
13. 其实数据一直是在这个路径下，只要有一个节点还保留，就会保存下去。
# ls /data/zookeeper/data/ myid version-2 zookeeper_server.pid
注意：一定要保证Server4/Server5的下面两个路径是空的，不然会出现，Server4/Server5识别的是之前的陈旧信息。
/data/zookeeper/data/version-2 /data/zookeeper/log/version-2
14. 说到这里，我们可以理解到，Zookeeper的全部数据，都是存放在下面两个路径中。如果需要做备份，可以直接在OS层面，做cp备份即可。
dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/log
衍生一个想法，就是如果想做跨Region，北京（主环境）到宁夏（容灾环境）的zookeeper的高可用怎么做呢？
我们可以考虑将北京的zookeeper的数据文件定期备份，并导入到宁夏的环境。
具体步骤：
<1. 在宁夏启动一个Zookeeper集群，并配置好，然后关闭zookeeper服务，清空掉数据文件夹。
<2. 在北京，通过脚本定期检查zookeeper各个节点状态，从一个运行健康的节点，定期备份数据到S3的一个bucket，为每个文件加上时间戳。
<3 through S3, Cross the Replication Region , synchronized to Ningxia.
<4. Ningxia Then, S3 is read from the backup file, and restored to the zookeeper disaster recovery.