ZooKeeper I think we should have heard a little, you may not be directly used in development, but commonly used Hadoop, HBase, Kafka, Dubbo and others have to use ZooKeeper. ZooKeeper in the end that played what role and why these frameworks, systems ZooKeeper need to use it, how should we use ZooKeeper in the development process, and whether there are alternatives ZooKeeper it. This article will focus on these questions, start with the following three aspects:
- Origin and role;
- Classic scenarios;
- alternatives.
1. origin and role
What ZooKeeper was originally designed? This is a research team from Yahoo's talking about. At that time, the researchers found inside Yahoo have a lot of need to rely on a distributed system components distributed coordination, but the presence of these components are often distributed single-point problem. So Yahoo will be the organization developed a common framework for coordination of distributed with no single point problem , that is ZooKeeper, the settlement of a single point , on the other hand, will be distributed coordination from distributed systems pulled out, allowing developers focus more on business logic.
Here on the "single-point problem" and "Distributed Coordination" were told.
1.1 single-point problem
Single-point problem (also known as single point of failure, Single Point of Failure, SPOF) refers to the system once the failure of the whole system will not work components. For example, the system will only be deployed on a machine A machine, if the machine A fails, the whole system will not work. In order to solve this problem, commonly used in a redundant fashion, increasing the number of machines, as long as the number of machines does not fail at the same time, the system will function properly.
ZooKeeper also solve the problem by means of a single point of the cluster. Seemingly by way of a cluster, the deployment of more than a few machines able to solve the problem of a single point. First, we need to issue a single point of refined into stateless of a single point and a single point of state issues . Here that the state may refer to the presence or absence of the concept of network protocol, HTTP stateless protocol, each request is independent, TCP stateful protocols, reliable transmission is completed relying on the recording state.
For simple stateless single point problem can be solved by means of the cluster, e.g., the proxy node forwards the message to do, it is no state, as shown below:
If the message forwarding X server is down, as well as server Y will continue to work normally, and the more the number of machines higher availability.
For state, not the same, and we will face distributed consistency problems. This issue alone is simply adding machine can not be solved. Common solutions include:
- Go to state: the stateless problem is removed, for example, to a reliable storage state of the DB;
- Master and slave: the state of the primary data processing do Master, the Master synchronization Slaver, such as MySQL from the master copy, the write operation processing Master, Slaver by binlog synchronization state.
Above brief introduction to MySQL solution, and ZooKeeper ZAB is the protocol used to achieve strong consistency. Distributed consensus on the issue, we will not continue to be described, everyone would remember first ZooKeeper to solve the problem of a single point, to ensure strong consistency , do not worry because there are ZooKeeper cluster machine downtime resulting in data inconsistencies.
Chat over a single point of issue, distributed coordination under the following chat, which is needed to ZooKeeper play a role.
1.2 Distributed Coordination
Distributed coordination, its meaning as its literal meaning, coordination in a distributed environment. For example, in a single case, if multiple threads at the same time want to make changes to a resource, such as inventory minus one. To ensure accuracy, we will lock on this resource. However, in the distributed case, if multiple machines want to make changes to a resource, how do we lock for this resource it? This time we need a coordinator arise. If you want to know the machine other machines have not locked, you do not need to communicate with other machines, simply coordinator of communication and can know the lock of resources, but also that the lock is managed by a coordinator.
You might say why should ZooKeeper as coordinator, why not choose a machine directly in the cluster. Yes, of course you can, but then you'll need to solve the problem of single point coordinator (as machine downtime coordinator). And because a distributed system mostly need such a coordinator component, and in order to reuse, and pulled it out, so there will be distributed coordination system . Therefore Dubbo, Kafak and other frameworks when implementing distributed directly to ZooKeeper use of them, so no need to repeat the components to achieve coordinator. And programmers we only need to focus on business logic to achieve in a distributed development.
Under summary, the role of ZooKeeper is provide no single point of coordination problems of distributed services.
2. The classic scenario
This section describes some of the ZooKeeper classic scenarios to see in the end how to use ZooKeeper, but first needed to ZooKeeper Data Model, the node characteristics, Watcher mechanisms have some knowledge, the following will be brief.
2.1 ZooKeeper basis
What is the data model, data can be understood as a storage format, e.g. Redis key is stored in the form of data, and ZooKeeper is a tree, as shown below:
Wherein the root node name is "/", the root node has two child nodes "A" and "B", "A" there are three child nodes "X", "Y", "Z". We can organize the tree nodes required, you can also store data on a node, the file system is a bit like the feeling.
With a basic understanding of data models, also need to know the characteristics of the next node, there are four nodes, are well understood:
- Persistent Node: Creating will persist unless the initiative to remove;
- Lasting order of nodes: its persistence and lasting nodes, but it has a sequential, plus a numeric suffix node name indicates the order in which they created, for example, 1,2,3,4 ...;
- Temporary node: If you create a session that node fails, then the node will be deleted;
- Temporal order nodes: node a temporary increase in the order concept.
Finally, knowing Watcher mechanism, is a sensing mechanism, we can monitor changes in the content of a data node, sub-node changes, once we monitor events, ZooKeeper will let us know.
ZooKeeper some basic introduction to, the following entry to the topic of this section.
2.2 Data Publish / Subscribe
By Watcher mechanism for data publish / subscribe. To configure the central scenario as an example. Publishing the configuration to a node on ZooKeeper and other machines can be achieved by dynamically update the configuration changes to a node on a monitor ZooKeeper. For example, when we put on the database configuration "/ configserver / app1 / database_config" node, and then let each machine to get database configuration information from the node at startup, and monitor changes in content nodes, configuration changes occur, retrieve configuration .
2.3 Naming Service
ZooKeeper by the node order to generate a globally unique ID. For example, a distributed task scheduling system, the production of globally unique ID for the task, as shown below:
After creating the sequence node unique ID of the task name can be obtained through the node name of the splice, for example, "type1-job-000000003".
2.4 Cluster Management
ZooKeeper by the temporary node and Watcher mechanism to monitor the operation state of the cluster, as shown below:
When new machines added to the cluster, create a temporary node under the "machines". When a machine goes down, temporary node will fail to be removed. By changes in child nodes under surveillance "machines", we can know the state of the cluster machine.
2.5 Distributed Lock
Distributed Lock, there are two categories, exclusive lock and a shared lock.
Exclusive lock during a transaction resource locking, does not allow other transactions to read and write. Through a temporary node will be able to represent a lock, as shown below:
The establishment of a "lock" node under "resourceA \ exclusive_lock" A node represents a resource on the exclusive lock. When multiple machines try to create a "lock" node under "exclusive_lock", there will only be a success can create, while others fail machines will listen "exclusive_lock" child of change. When the lock is released actively or down, "lock" node will be removed, other machines will be notified and continue to try to create a "lock" node to acquire the lock.
Shared lock during a transaction resource locking, allowing other transactions for reading. Is still represented by a node locks difference is that the need to distinguish between the read and write operations, read and write the data content may be represented by a node, for example, R represents a read operation, W indicates a write operation. As shown below:
Read and write operations are called to create a "lock" in the "shared_lock" node interim order of the nodes , but different data content. For read operations, if the number is less than your own child nodes are read request, thinks he successfully obtained a lock, you can read, if a small number of nodes included in the write operation, it is necessary to wait, listen son "shared_lock" of node changes. For write operations, if they are not the smallest number of child nodes, into the waiting, listening "shared_lock" child of change.
3. alternatives
Section 2, we learned about ZooKeeper there are many scenarios in a distributed environment, it is not necessary to use ZooKeeper to achieve distributed lock, cluster management and other functions? Of course not, there are other techniques to choose from. For example Redis also can be used for distributed coordination service, said on the second of those scenes, are able to achieve, but because of their different data model requires a different design.
That we use ZooKeeper or Redis it? This needs to be decided depending on the scene, we contrast down at the advantages and disadvantages of both the distributed lock scene:
- Redis: Support for high concurrency, release the lock operation, can not guarantee 100% data consistency problems may occur (rare);
- ZooKeeper: lock model robust, strong consistency, but the frequent application lock, release the lock on the operating pressure ZooKeeper cluster larger.
Redis data consistency is better ZooKeeper, and ZooKeeper for high concurrency support as good as Redis.
The last would like to introduce, ectd, this is a highly available key-value storage system, use the Go write, copy, ensure strong consistency by Raft algorithm processing log is used by Kubernetes systems. Compared with ZooKeeper words, there are many different technical details, I'll give some broad, ZooKeeper is a Java-based implementation, naturally, has the disadvantage of Java, such as GC pauses, but ectd no problem GC pauses. ZooKeeper have a good Java client library, it also provides client libraries in other languages, ectd as a rising star, there is no good Java client library, it can only be called via HTTP.
Redis listed above and ectd only to help readers unobscured, not limited to ZooKeeper. And how to carry out specific technology selection, they still lack experience, not to say.
4. Reference
- "From Paxos to ZooKeeper"
- Why ZooKeeper
I like the article junior partner, you can sweep under the code number of public concern me: "Pinch grass child"