Symptoms:
Use hbase shell connection reported the following questions:
2019-10-09 10:37:18,855 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts
2019-10-09 10:37:18,856 WARN [main] zookeeper.ZKUtil: hconnection-0x6ef784bf0x0, quorum=xxx:2181,xxx:2181,xxx:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid)
log log which hbase given as follows:
2019-10-09 09:26:58,701 WARN [regionserver/xxx/192.168.1.8:16020-longCompactions-1569222224980-SendThread(xxx:2181)] zookeeper.ClientCnxn: Session 0x0 for server xxx/192.168.1.24:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
Resolution process:
Issue on appeal by the phenomenon can be found because of a problem zookeeper, the first attempt to look at whether the zookeeper hang.
1, using the telnet host or ip 2181 test connection
# telnet xxx 2181
Trying 192.168.1.23...
Connected to xxx.
Escape character is '^]'.
Connection closed by foreign host.
Connection is not found in the past, the remote server or remote procedure closed the connection
2, connected to the server node to view zookeeper socket connections
: ~ $ Netstat -anl | grep 2181 | grep -i '192.168.1.7' | grep ESTABLISHED | wc -l
1
: ~ $ netstat -anl | grep 2181 | grep -i '192.168.1.8' | grep ESTABLISHED | wc - l
60
192.168.1.8 machine even if a server error hbase appeal can be found to keep the machine in the current session of zookeeper node is 60, which is far from the limit of the system
3, the modified hbase connection limit zookeeper
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value> 300 </ value> # default is 30, after modification, reboot regioserver, but useless
property>
4, zoo.cfg file under the modified zookeeper
maxClientCnxns = # 60 this value with the number ESTABLISHED connection just to see exactly the same cancel comments, revised to 150, reboot zookeeper
problem solved