1. Overview
This document is a guide for developers who wish to create distributed applications utilizing the ZooKeeper coordination service. It contains conceptual and practical content.
The first four parts of this guide provide a high-level discussion of various ZooKeeper concepts. These are all necessary to understand how ZooKeeper works and how to use them. It does not contain source code, but it does address issues related to distributed computing.
Much of the information in this document has been written as stand-alone reference. However, before starting your first ZooKeeper application, you should at least read the chapters on the ZooKeeper data model and ZooKeeper basic operations. Also, the Simple Programming Examples chapter will help you understand the basic structure of a ZooKeeper client application.
2. ZooKeeper data model
ZooKeeper has a hierarchical namespace, much like a distributed file system. The only difference is that each node in the namespace can have data associated with it as well as child objects. It's like having a special filesystem that allows a file to be a directory at the same time. Paths are usually represented as canonical, absolute, slash-separated paths, with no relative paths. Any unicode character can be used in paths subject to the following constraints:
- The null character (\u0000) cannot be part of a pathname. (This causes problems with C-bound clients.)
- The following characters cannot be used because they display poorly, or are rendered in confusing ways: \u0001 - \u0019 and \u007F - \u009F.
- The following characters are not allowed: \ud800 - uF8FFF, \uFFF0 - uFFFF.
- The "." character can be used as part of another name, but "." and ".." alone cannot be used to indicate nodes on a path, because ZooKeeper does not support relative paths. For example the following are invalid: "/a/b/./c" or "/a/b/../c".
- The token "zookeeper" is reserved
2.1 znodes
Each node in the ZooKeeper directory tree is called a znode. Znodes maintain a stat structure that contains version numbers of data changes, acl (Access Control List) changes. The stat structure also has timestamps. The presence of the version number and timestamp allows ZooKeeper to validate the cache and coordinate updates. Every time the data of the znode changes, the version number is incremented. For example, whenever the client gets data, it also gets the version information of the data. When a client performs an update or delete, it must provide the data version of the znode it is changing. If the version it provides doesn't match the actual version's data, the update will fail. (This behavior can be overridden. For more information, see...)
Note: In a distributed application, a word node can refer to a generic host, server, cluster member, client process, etc. In the ZooKeeper documentation, znodes refer to data nodes. Servers are the machines that make up the ZooKeeper service; quorum peers are the servers that make up the whole; clients are any hosts or processes that use the ZooKeeper service.
The znode is the main abstraction that programmers need to be aware of. Znodes have several features worth mentioning here.
2.1.1Watches
Clients can set watches on znodes. Changes to that znode trigger the watch, which then clears the watch. ZooKeeper will send a notification to the client when the watch fires.
2.1.2 Data Access
Data on each node is read and written atomically. A read gets all the data bytes associated with the znode, a write replaces all data. Each node has an access control list (ACL) that restricts who can do what.
ZooKeeper is not designed for general purpose databases or large object stores. Instead, it is used to manage coordination data. These data can be configuration information, status information, etc. A common characteristic of all forms of coordination data is that they are relatively small: measured in kilobytes. The ZooKeeper client and server implementations have sanity checks to ensure that the znode has less than 1M of data, but the actual data should be much less than 1M. Operating on relatively large data can cause some operations to take more time than others, and can affect the latency of some operations because it takes extra time to transfer more data over the network to the storage medium. If a large amount of data storage is required, the usual pattern for handling this data is to store it in a mass storage system such as NFS or HDFS, and then store a pointer to the storage location in ZooKeeper.
2.1.3 Ephemeral Nodes
ZooKeeper also has the concept of ephemeral nodes. These znodes exist as long as the session that created the znode is active. When the session ends, the znode is deleted. Due to this behavior, temporary znodes are not allowed to have children.2.1.4 Sequence Nodes -- Unique Naming
When creating a znode, you can also request ZooKeeper to append a monotonically increasing counter to the end of the path. This counter is unique to the parent znode. The format of the counter is %010d - i.e. 10 digits and 0 (zero) padding (the counter is formatted this way to simplify sorting), i.e. "<path>0000000001". See the Queue Recipe chapter for an example use of this feature. Note: The counter used to store the next sequence number is a signed integer (4 bytes) maintained by the parent node, the counter overflows when it increments above 2147483647 (the node name at this point is "<path>-2147483647") .
2.2 Time in ZooKeeper
ZooKeeper tracks time in a number of ways:
Zxid
Every change to the ZooKeeper state is marked with a znode in the form of zxid (ZooKeeper Transaction Id). zxid represents the total ordering . Each change will have a unique zxid, if zxid1 is less than zxid2, then zxid1 happens before zxid2.
Version numbers
Every change to a node causes some version number of that node to be incremented. The three version numbers are version (the number of changes to the znode's data), cversion (the number of changes to the znode's children), and aversion (the number of changes to the znode's ACL).
Ticks
When using multi-server ZooKeeper, servers use ticks (heartbeats ) to define the timing of events such as state upload time, session timeout time, connection timeout between server peers, etc. The minimum session timeout is 2 times the heartbeat time ; if the client requested session timeout is lower than the minimum session timeout, the server will notify the client that the session timeout is actually the minimum session timeout.
Real time
ZooKeeper doesn't use real-time or clock time at all, other than putting timestamps into the stat structure when znodes are created and znodes are modified.
2.3 ZooKeeper Stat Structure
The Stat structure for each znode in ZooKeeper consists of the following fields:
czxid
The zxid of the change that caused this znode to be created.
mzxid
The zxid of the change that last modified this znode.
pzxid
The zxid of the change that last modified children of this znode.
ctime
The time in milliseconds from epoch when this znode was created.
mtime
The time in milliseconds from epoch when this znode was last modified.
version
The number of changes to the data of this znode.
cversion
The number of changes to the children of this znode.
aversion
The number of changes to the ACL of this znode.
ephemeralOwner
The session id of the owner of this znode if the znode is an ephemeral node. If it is not an ephemeral node, it will be zero.
dataLength
The length of the data field of this znode.
numChildren
The number of children of this znode.
3 ZooKeeper Sessions
ZooKeeper客户端通过使用语言绑定创建服务句柄来建立与ZooKeeper服务的会话。一旦创建，句柄的状态就是CONNECTING，当客户端库尝试连接到构成ZooKeeper服务的服务器之一时，此时状态将切换到CONNECTED。在正常操作过程中将处于这两种状态之一。如果发生不可恢复的错误，例如会话过期或身份验证失败，或者应用程序明确关闭了句柄，则句柄将移至CLOSED状态。下图显示了ZooKeeper客户端可能的状态转换：
要创建客户端会话，应用程序代码必须提供一个连接字符串，其中包含逗号分隔的host：port对列表，每个对应于一个ZooKeeper服务器（例如“127.0.0.1:4545”或“127.0.0.1:3000,127.0.0.1 ：3001,127.0.0.1：3002" ）。 ZooKeeper客户端库将选择一个任意的服务器并尝试连接到它。如果此连接失败，或者客户端因任何原因与服务器断开连接，则客户端将自动尝试列表中的下一台服务器，直到（重新）建立连接。
在3.2.0版本中中添加功能：可选的“chroot”后缀也可以附加到连接字符串。这将在解释与该根相关的所有路径时运行客户端命令（类似于unix chroot命令）。如果使用该示例，其格式如下：“127.0.0.1:4545/app/a”或“127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002/app/a”，其中客户端将认为所有路径都是相对于此根目录的 -例如在“/ foo / bar”目录下执行获取、设置等操作，将导致在“/ app / a / foo / bar”目录中执行操作）。此功能在多租户环境中特别有用，其中特定ZooKeeper服务的每个用户可以设置不同的根。这使得重复使用更加简单，因为每个用户可以将他/她的应用程序编码为“/”，而实际位置（比如/ app / a）可以在部署时确定。
当客户端获得zookeeper服务的句柄时，zookeeper创建一个zookeeper session，表示为一个64位数字，它分配给客户端。如果客户端连接到不同的zookeeper服务器，它将发送session ID作为连接握手的一部分。作为安全措施，服务器会为session id创建一个密码，这个密码用于被其他的zookeeper服务器验证。当客户端重新建立session时，密码将会和session id一起发送到客户端。
zookeeper客户端库调用创建zookeeper session的参数之一是以毫秒为单位的session timeout。客户端发送一个期望timeout，服务器响应它可以允许客户端的session timeout。当前的实现要求session timeout至少为ticktime(心跳时间)的2倍（在服务器配置中设置），最大为ticktime的20倍。
当客户端（session）从zk服务集群分区时，它将开始搜索会话创建期间指定的服务器列表。最终，当客户端和至少一个服务器之间的连接重新建立时，会话将再次转换到“connected”状态（如果在会话超时时间内重新连接），或者它将转换到“expired”状态（如果在会话超时后重新连接）。不建议创建新的会话对象（c绑定中的新zookeeper.class或zookeeper句柄）以断开连接。zk客户端库将为您处理重新连接。特别是我们在客户端库中内置了启发式函数来处理诸如“herd effect”等事情......只有在您被通知会话过期时才强制性的创建新会话。
会话过期由zookeeper集群本身管理，而不是由客户端管理。当zk客户端与集群建立会话时，它会提供上面详述的“timeout”。该值由集群用于确定客户端的会话何时到期。当群集在指定的会话超时期间内没有从客户端收到响应（即，无心跳）时，会使会话到期。在会话到期时，集群将删除该会话拥有的任何/所有临时节点，并立即通知任何/所有连接的客户端该更改（任何观察这些znode的客户端）。客户端将保持断开连接状态，直到与群集重新建立tcp连接，此时过期会话的观察者将收到“会话过期”通知。
过期会话的观察者看到过期会话的状态转换示例：
'connected' : session is established and client is communicating with cluster (client/server communication is operating properly)
.... client is partitioned from the cluster
'disconnected' : client has lost connectivity with the cluster
.... time elapses, after 'timeout' period the cluster expires the session, nothing is seen by client as it is disconnected from cluster
.... time elapses, the client regains network level connectivity with the cluster
'expired' : eventually the client reconnects to the cluster, it is then notified of the expiration
default watcher是zookeeper会话建立连接的另一个参数。当客户端发生任何状态变化时，通知这个观察者。例如，如果客户端失去与服务器的连接，客户端将被通知，或者如果客户端的会话过期等等，该观察者应该考虑初始状态为断开（即，在客户端库发送任何状态改变事件到观察者之前）。在新连接的情况下，发送给观察者的第一个事件通常是会话连接事件。
该会话通过客户端发送的请求保持活动状态。如果会话空闲一段时间会超时，客户端将发送ping请求以保持会话活动。此ping请求不仅使zookeeper服务器知道客户端仍处于活动状态，还允许客户端验证其与zookeeper服务器的连接仍处于活动状态。ping的时间设置足够保守，以确保有合理的时间检测死连接并重新连接到新服务器。
一旦与服务器的连接成功建立（connected），当执行同步或异步操作并符合以下操作之一时，客户端在发生连接丢失基本上有两种情况（java中的异常 - 请参阅绑定特定细节的api文档）：
1、应用程序没有在alive/valid的会话上调用操作
2、当存在对该服务器的挂起操作时，例如，存在挂起的异步调用时，zookeeper客户端与服务器断开连接
在3.2.0中添加的新功能 - sessionmovedexception。有一个内部异常被称为sessionmovedexception通常不会被客户端识别。发生此异常的原因是，在不同服务器上重新建立会话的连接上收到请求。导致此错误的正常原因是客户端向服务器发送请求，但网络数据包延迟，因此客户端超时并连接到新的服务器。当延迟的数据包到达第一台服务器时，第一台服务器检测到会话已经移动，并关闭客户端连接。客户端通常不会看到这个错误，因为他们没有从这些旧连接中读取。（旧连接通常是关闭的。）可以看到这种情况的一种情况是两个客户端尝试使用保存的会话标识和密码重新建立相同的连接。其中一个客户端将重新建立连接，第二个客户端将断开连接（导致服务器对无限期地尝试重新建立连接/会话）。
4、ZooKeeper Watches
All read operations in zookeeper - getdata(), getchildren() and exists() - have the option to set a watch or not. Here is zookeeper's definition of watch: watch event is a one-time trigger, sent to the client that sets the watch, and triggers the watch event when the data of the set watch changes. There are three key points to consider in this definition of watc:1. One-time trigger:
When the data changes, a watch event will be sent to the client. For example, if a client executes getdata("/znode1", true), and subsequently changes or deletes data for /znode1, the client will get watch events for /znode1. If /znode1 changes again, the watch event will not be sent unless the client does another read that sets a new watch.
2. Send to the client
This means that an event is being sent to the client, but the watch event may not reach the client that set the watch until the success return code of the change operation reaches the client that initiated the change. The watch event is sent asynchronously to the client. zookeeper provides an ordering guarantee: a client will never receive changes to the watch it sets until it receives a watch event for the first time. Network latency or other factors may cause different clients to receive watch events at different times and return update codes. The point is that everything will be seen in a consistent order by different customers.
3. Set the data of the watch
这是指节点可以改变的不同方式。想象zookeeper维护两个watch列表是有帮助的：data watches and child watches。getdata（）和exists（）设置data watchs。getchildren（）设置child watches。或者，根据返回的数据类型，可能会帮助您思考设置watch。getdata（）和exists（）返回有关节点数据的信息，而getchildren（）返回一个子节点列表。因此，setdata（）将触发设置的znode的data watch假设该设置成功）。一个成功的create（）将触发一个对于正在创建的znode的data watch，以及正在创建的znode的parent node的child watch。一个成功的delete（）会对正在被删除的znode同时触发一个data watch和一个child watch（因为不能有更多的子节点），也会触发正在删除的znode的parent node的child watch。
watch保存在客户端连接的那台zookeeper服务器上。这使得watch可以被轻量级的设置，维护和发送。当客户端连接到新的服务器时，watch将会被任何session event触发。在与服务器断开连接时不会收到watch。当客户重新连接时，任何先前注册的watch将被重新注册并在需要时触发。一般而言，这一切都是透明的。有一种情况下watch可能会丢失：对一个还未创建的znode设置一个是否存在的watch， 如果在断开连接时znode被创建然后删除，那么这个watch被忽略。
3.1 watch的语义
We can set the watch with three calls to read the zookeeper state: exists, getdata and getchildren. The following list details the events that a watch can fire and the calls to enable those events:
Created event:
Enabled with a call to exists.
Deleted event:
Enabled with a call to exists, getData, and getChildren.
Changed event:
Enabled with a call to exists and getData.
Child event:
Enabled with a call to getChildren.
Regarding watches, zookeeper has these guarantees:
- Watches are ordered relative to other events, other watches and async accordingly. The zookeeper client library ensures that everything is scheduled in order.
- The client will see watch events for the znode it is monitoring before seeing new data corresponding to that znode.
- The order of listening events from zookeeper corresponds to the order of updates seen by the zookeeper service
3.3 Things to remember about watches
A watch is a one-time trigger; if you receive a watch event and want to be informed about future changes, you must set a new watch.
Because watches are one-time triggers, and there is a delay between getting an event and sending a new request to get a new watch, it's impossible to reliably see every change that happens to a znode in zookeeper. Need to consider handling the case where the client znode is changed multiple times between getting the event and setting a new watch. (You probably don't care, but at least need to be aware that it can happen.)
当您从服务器断开连接时（例如，服务器出现故障时），在连接重新建立之前，您将不会收到任何watch。出于这个原因，session event被发送到所有的watch handlers。利用session events进入安全模式：断开连接时不会收到事件，因此您的流程应该在该模式下保守行事。
4、zookeeper访问控制--ACLs
zookeeper使用acls来控制对其znode的访问。acl实现与unix文件访问权限非常相似：后者使用权限位来允许/禁止针对节点的各种操作以及这些位适用的范围。与标准的Unix权限不同，zookeeper节点不受用户（文件所有者），group和world（other）的三个标准作用域的限制。zookeeper没有znode所有者的概念。相反，acl指定了ids的集合以及与这些ID相关联的权限。
还要注意，acl仅适用于具体的znode。特别是它不适用于具体znode的子节点。例如，如果/ app只能通过ip：172.16.16.1读取，那么/ app / status是所有ip都可读的，任何人都可以读/ app / status;acls不是递归的。
zookeeper支持可插入的身份验证schemes。Ids是使用scheme:id形式指定的，其中scheme是id对应的认证方案。例如，ip：172.16.16.1是地址为172.16.16.1的主机的ID。
当一个客户端连接到zookeeper并进行身份验证时，zookeeper会将与客户端对应的所有id与客户端连接相关联。当客户端尝试访问节点时，会根据znodes的ACL检查这些ID。acls由（scheme:expression, perms）形式的数据组成。expression的格式是特定于该scheme的。例如，(ip:19.22.0.0/16, READ)为具有以19.22开头的ip地址的任何客户端提供读取权限。
4.1 ACL Permissions
zookeeper支持以下权限：
CREATE: you can create a child node
READ: you can get data from a node and list its children.
WRITE: you can set data for a node
DELETE: you can delete a child node
ADMIN: you can set permissions
create和delete权限是对write权限的更细粒度的访问控制权限。create和delete的情况如下：
下面两句段话，我没有理解是什么意思，原文先贴在这里，后续再理解。
You want A to be able to do a set on a ZooKeeper node, but not be able to CREATE or DELETE children.
CREATE without DELETE: clients create requests by creating ZooKeeper nodes in a parent directory. You want all clients to be able to add, but only request processor can delete. (This is kind of like the APPEND permission for files.)
此外，由于zookeeper没有文件所有者的概念，所以才有了admin权限。在某种意义上，admin权限指定entity为所有者。zookeeper不支持LOOKUP权限（在目录上的执行权限位允许您查找，即使您不能列出目录）。每个人都隐含有查询权限。这允许你统计一个节点，但不能执行更多的操作。（问题是，如果您想在不存在的节点上调用zoo_exists（），则没有权限检查。）
4.1.1 内置acl方案
zookeeper有以下内置方案：
world has a single id, anyone, that represents anyone.
auth doesn't use any id, represents any authenticated user.
digest uses a username:password string to generate MD5 hash which is then used as an ACL ID identity. Authentication is done by sending the username:password in clear text. When used in the ACL the expression will be the username:base64 encoded SHA1 password digest.
ip uses the client host IP as an ACL ID identity. The ACL expression is of the form addr/bits where the most significant bits of addr are matched against the most significant bits of the client host IP.
5、可插入式zookeeper认证
zookeeper运行在各种不同的环境中，具有各种不同的身份验证方案，因此它具有完全可插入的身份验证框架。内置身份验证方案也使用可插入身份验证框架。
要了解身份验证框架是如何工作的，首先您必须了解两个主要的身份验证操作。框架首先必须验证客户端。这通常在客户端连接到服务器后立即完成，并且包括验证从客户端发送或收集的信息并将其与connection相关联。框架处理的第二个操作是在acl中查找与客户端对应的条目。acl条目是<idspec，permissions>对。idspec可能是与连接相关的验证信息的简单字符串匹配，也可能是针对该信息评估的表达式。这取决于采用的身份验证插件的实现情况。这里是一个认证插件必须实现的接口：
public interface AuthenticationProvider {
String getScheme();
KeeperException.Code handleAuthentication(ServerCnxn cnxn, byte authData[]);
boolean isValid(String id);
boolean matches(String id, String aclExpr);
boolean isAuthenticated();
}
第一个方法getscheme返回标识该插件的字符串。因为我们支持多种身份验证方法，身份验证凭证或idspec将始终以scheme：为前缀。zookeeper服务器使用由认证插件的getScheme()方法返回的scheme来确定该方案适用于哪个ID。
当客户端发送认证信息以与连接相关联时，handleauthentication被调用。客户端指定信息对应的scheme。zookeeper服务器将信息传递给对应的认证插件，这个对应的认证插件的getcheme()返回值匹配客户端传递的shceme。如果handleauthentication的实现者确定信息错误，它通常会返回一个err，否则它会使用cnxn.getAuthInfo().add(new Id(getScheme(), data))将信息与连接相关联。
身份验证插件涉及设置和使用acls两部分。当为znode设置acl时，zookeeper服务器会将entry的id部分传递给isvalid(string id）方法。由插件来验证该ID是否具有正确的格式。例如，ip：172.16.0.0/16是一个有效的id，但ip：host.com不是。如果新的acl包含“auth” entry，则使用isauthenticated()方法来查看与该connection关联的scheme的认证信息是否应该添加到acl。一些schemes不应包含在auth中。例如，如果scheme指定为auth，则客户端的IP地址不应该添加到acl的id。
zookeeper在检查acl时调用matches(String id, String aclExpr)方法。需要将客户端的认证信息与相关的acl表项进行匹配。为了找到适用于客户端的条目，zookeeper服务器将查找对应scheme的每个条目，并且如果客户端传了相应scheme的认证信息，则调用matches(String id, String aclExpr)方法时会将id设置为之前通过handleauthentication()方法添加到connection的信息，并将aclexpr设置为acl条目的id。身份验证插件根据其自己的逻辑和匹配的scheme共同确定id是否包含在aclexpr中。
有两个内置认证插件：ip和digest。额外的插件可以使用系统属性进行添加。在启动时，zookeeper服务器将查找以“zookeeper.authprovider”开头的系统属性。并将这些属性的值解释为认证插件的类名称。可以使用-dzookeeeper.authprovider.x = com.f.myauth或在服务器配置文件中添加以下条目来设置这些属性：
authProvider.1=com.f.MyAuth authProvider.2=com.f.MyAuth2
应小心谨慎，以确保该属性的后缀是唯一的。如果有重复项，例如-dzookeeeper.authprovider.x = com.f.myauth -dzookeeper.authprovider.x = com.f.myauth2，则只会使用一个。所有服务器也必须定义相同的插件，否则使用插件提供的身份验证方案的客户端将无法连接到某些服务器。
6、一致性保证
zookeeper是一款高性能，可扩展的服务。读取和写入操作都很快，但读取速度比写入速度快。原因是在读取的情况下，zookeeper可以提供较旧的数据，之所以能使用较旧的数据又是由于zookeeper的一致性保证：
顺序一致性
来自客户端的更新将按照它们发送的顺序执行。
原子性
更新要么成功要么失败
统一系统映像
无论连接到哪个服务器，客户端都会看到相同的服务视图。
可靠性
一旦应用了更新，它就会一直持续到客户覆盖更新为止。这个保证有两个推论：
1、如果客户端获得成功的返回码，则更新将被应用。在某些故障（通信错误，超时等）时，客户端将不知道更新是否已应用。虽然我们采取措施尽量减少故障，但这个保证仅在返回成功码时适用。（这在paxos中称为单调性条件。）
2、当从服务器故障中恢复时，客户端通过读取请求或成功更新所看到的任何更新都不会回滚。
及时性
保证系统的客户端视图在一定的时间范围内（大约几十秒）是最新的。客户可以在此范围内看到任何系统更改，或者客户端将检测到服务中断。
基于这些一致性保证很容易构建更高级别的功能，例如仅在zookeeper客户端（不需要zookeeper）构建一些功能例如，leader选举， barriers, 队列和读/写可撤销锁。
7、Java binding
有两个包构成了zookeeper java绑定：org.apache.zookeeper和org.apache.zookeeper.data。组成zookeeper的其余包在内部使用或者是服务器实现的一部分。org.apache.zookeeper.data包由生成的classes组成，这些类仅用作容器。
zookeeper java客户端使用的主类是zookeeper类。它的两个构造函数只有一个可选的会话ID和密码。zookeeper支持跨进程实例的会话恢复。一个Java程序可能会将其会话标识和密码保存到稳定的存储中，用于重新启动并恢复该程序之前的实例中使用的会话。
当创建zookeeper对象时，还会创建两个线程：一个io线程和一个event线程。所有io都发生在io线程上（使用java nio）。所有的事件回调都发生在event线程上。会话维持（例如重新连接到zookeeper服务器并保持心跳）在io线程上完成。同步方法的响应也在io线程中处理。所有对异步方法和监视事件的响应都在event线程上处理。有几件事要注意这个设计的结果：
所有异步调用和观察者回调的完成都将按顺序进行。调用者可以执行他们希望的任何处理，但在此期间不会处理其他回调。
回调不会阻止io线程的处理或同步调用的处理。
同步调用可能不会以正确的顺序返回。例如，假设客户端执行以下处理：发出节点/ a的异步读取，并将watch设置为true，然后在读取的完成回调中执行/ a的同步读取。（也许不是很好的做法，但也不是非法的，它只是一个简单的例子。）请注意，如果异步读取和同步读取之间节点/a 发生了变化，假设/a节点在同步读取的响应返回之前发生了变化，则客户端库将接收到监视事件，但由于完成回调阻塞了event队列，在监视事件被处理之前，同步读取将返回/a节点的新值。
最后，与关机相关的规则很简单：一旦zookeeper对象关闭或接收到致命事件（session_expired和auth_failed），则zookeeper对象将变为无效。结束时，两个线程关闭，zookeeper句柄的任何进一步访问都是未定义的行为，应该避免