Introduction to Zookeeper
Zookeeper is a distributed service framework and a sub-project of Apache Hadoop. It is mainly used to solve some data management problems often encountered in distributed applications, such as: unified naming service, state synchronization service, cluster management, distributed Management of application configuration items, etc. The previous explanation is a bit official. Simply put, zookeeper=file system+monitor notification mechanism.
Note: The connection between the client and the server is based on the TCP long connection , the bottom layer is by default through the NIO method of java, and the netty implementation method can also be configured. The client side connects to the default port 2181 on the server side, which is the session. Starting from the first connection establishment, the client begins the life cycle of the session, and the client requests a ping packet from the server. Each session can set a timeout period.
zookeeper data structure
The name space provided by zookkeeper is very similar to the standard file system, which is stored in the form of key-value. The name key is a series of path elements separated by a slash /. Each node in the zookeeper name space is identified by a path.
Related CAP theory
The CAP theory points out that for a distributed computing system, it is impossible to satisfy the following three points at the same time:
- Consistency : In a distributed environment, consistency refers to whether the data can be consistent among multiple copies, which is equivalent to all nodes accessing the same copy of the latest data. Under the requirement of consistency, when a system performs an update operation in a consistent state of data, it should ensure that the data of the system is still in a consistent state.
-
Availability: The correct response can be obtained for each request, but the data obtained is not guaranteed to be the latest data.
-
Partition fault tolerance: When a distributed system encounters any network partition failure, it still needs to be able to provide external services that meet the consistency and availability, unless the entire network environment fails.
A distributed system can only satisfy two of the three items: Consistency, Availability, and Partition tolerance at the same time.
Of these three basic requirements, only two of them can be met at the same time. P is a must, so you can only choose between CP and AP. Zookeeper guarantees CP, compared to the implementation of the registration center eruka in the spring cloud system It's AP.
BASE theory
BASE is the abbreviation of the three phrases Basically Available, Soft-state and Eventually Consistent.
-
Basically usable: When the distributed system fails, it is allowed to lose part of the availability (service degradation, page degradation).
-
Soft state: Allows an intermediate state in a distributed system. And the intermediate state does not affect the availability of the system. The intermediate state here refers to the eventual consistency in which data updates between different data replications (data backup nodes) can be delayed.
- Eventual consistency: Data replications reach consistency over a period of time.
BASE theory is the result of a trade-off between consistency and availability in CAP. The core idea of the theory is: we cannot achieve strong consistency, but each application can adopt an appropriate method to make the system reach Eventual consistency.
Four types of znode
-
PERSISTENT-Persistent Directory Node
After the client disconnects from zookeeper, the node still exists
-
PERSISTENT_SEQUENTIAL-Persistent Sequence Number Directory Node
After the client is disconnected from zookeeper, the node still exists, but Zookeeper gives the node name a sequential number
-
EPHEMERAL-temporary directory node
After the client disconnects from zookeeper, the node is deleted
-
EPHEMERAL_SEQUENTIAL-Temporary Sequence Numbering Directory Node
After the client disconnects from zookeeper, the node is deleted, but Zookeeper gives the node name a sequential number
Zookeeper node characteristics
1. The key name of the node at the same level is unique
2. When creating a node, you must bring the full path
3. The session is closed and the temporary node is cleared
4. Automatically create sequence nodes
5. Watch mechanism to monitor node changes
The event monitoring mechanism is similar to the observer mode. The watch process is that the client registers a watcher on a node path of the server, and the client also stores a specific watcher. When the node data or child nodes change, the server notifies the client At the end, the client performs callback processing. Special note: After the monitoring event is triggered once, the event becomes invalid.
Tip: Refer to the get command to monitor the use of watch in the chapter of common commands. The following chapters will introduce the principle of watch implementation in detail.
6. The delete command can only delete layer by layer
Tip : The new version can be deleted recursively through the deleteall command.
With the above-mentioned many node characteristics, zookeeper can not develop different classic application scenarios, such as:
- Data publish/subscribe
- Load balancing
- Distributed coordination/notification
- Cluster management
- master management
- Distributed lock
Utilizing the unique feature of zookeeper's peer nodes, multiple users create temporary child nodes under a certain menu at the same time, and successfully create a distributed lock. Other users who have not obtained the lock register a watcher that changes the child node under this menu. Listen to the event in order to regain the lock.
- Distributed queue
Zookeeper data synchronization process
In Zookeeper, ZAB protocol is mainly used to achieve distributed data consistency.
The ZAB agreement is divided into two parts:
- News broadcast
- Crash recovery
News broadcast
Zookeeper uses a single main process Leader to receive and process all transaction requests from the client (even if the follower node receives the request, it will be forwarded to the leader, and then the leader will send the transaction request to all the follower nodes) , and adopts the atomic broadcast protocol of the ZAB protocol , Broadcast the transaction request to all Follower nodes as a Proposal proposal. When more than half of the Follower servers in the cluster give correct ACK feedback, then the Leader will send a commit message to all Follower servers again to submit the proposal. This process can be referred to as 2pc transaction submission for short. The whole process can refer to the following figure. Note that the Observer node is only responsible for synchronizing Leader data and does not participate in the 2PC data synchronization process.
Crash recovery
Under normal circumstances, the message broadcast can work well, but once the Leader server crashes, or the Leader server loses communication with more than half of the Followers due to network principles, it will enter the crash recovery mode and a new Leader server needs to be elected. In this process, there may be two kinds of hidden dangers of data inconsistency, which need to be avoided by the characteristics of the ZAB protocol.
- After the leader server sends the message commit, it crashes immediately
- The Leader server crashed immediately after making a proposal
The recovery model of the ZAB protocol uses the following strategies:
- Elect the node with the largest zxid as the new leader
- The new leader processes the uncommitted messages in the transaction log