Zookeeper encounter a variety of pit

Copyright: original works, posted please indicate the source! https://blog.csdn.net/sa19861211/article/details/90300670

Problem Description:

Zookeeper cluster in production, five nodes zk, 15 minutes kept the internal elections, see section log some states become flower or leading, and after a while went looking up. Later, most of the dubbo reconnection of services are coming up after zk cluster stable, but there are several sets of services are no longer reconnection up, you must restart the application, as if the client reconnection is also a problem.

problem causes:

error log when the problem occurs:

fsync-ing the write ahead log in SyncThread:0 took 29761ms which will
adversely effect operation latency. See the ZooKeeper troubleshooting
guide

This is because the leader in data synchronization when the brush plate, disk IO latency is too large, exceeding the timeout session of the cause. This is a classic problem Zookeeper.