One of the nodes cannot connect to zookeeper when Hbase is started (maxClientCnxns)

When installing the hbase cluster, one of the nodes reported the following error when starting the cluster:
2016-09-09 15:32:57,749 INFO [main-SendThread(madwx31:2181)]

zookeeper.ClientCnxn: Socket connection established to madwx31/192.168.32.31:2181, initiating session
2016-09-09 15:32:57,749 INFO  [main-SendThread(madwx31:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-09-09 15:32:59,379 INFO  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Opening socket connection to server madwx51/192.168.32.51:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-09 15:32:59,380 INFO  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Socket connection established to madwx51/192.168.32.51:2181, initiating session
2016-09-09 15:32:59,380 WARN  [main-SendThread(madwx51:2181)] zookeeper.ClientCnxn: Session 0x0 for server madwx51/192.168.32.51:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO (ClientCnxnSocketNIO.java:68)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-09-09 15:32:59,481 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2016-09-09 15:32:59,481 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2422)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232)
	at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
	at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2432)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: master:160000x0, quorum=madwx51:2181,madwx41:2181,madwx31:2181, baseZNode=/hbase Unexpected KeeperException creating base node
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:206)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:187)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:585)
	at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:379)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2415)
	... 5 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:575)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:554)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1204)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1182)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:194)
	... 13 more

At first glance, it probably means that the zookeeper cannot be connected, but everything is normal when checking the zookeeper service, but when viewing the log file of the zookeeper, it is found that the error of too many connections from host - max is 10 is reported, which is obviously the number of connections that hbase connects to the zookeeper. There are too many, so add the following configuration to the zoo.cfg configuration file of zookeeper:
maxClientCnxns=300
and then restart zookeeper, the hbase problem is solved.


maxClientCnxns=300 The
default number of connections used by zookeeper for each client IP is 10, and there are often not enough connections. Modifying the number of connections seems to only support the modification of the zoo.cfg configuration file, so it needs to restart the zookeeper to take effect.
zoo.cfg:
maxClientCnxns=300
otherwise the following error will be reported: 2011-10-28 09:39:44,856 – WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:5858:NIOServerCnxn$Factory@253] – Too many connections from / 192.168.*.* – max is 10