Java——"Interview Questions——Zookeeper"

Literal translation: The literal translation of the name is animal administrator. Animal refers to distributed software such as Hadoop. The three characters of administrator reflect the characteristics of ZooKeeper: maintenance, coordination, management, and monitoring.

Brief description: Some software you want to make into a cluster or distributed, you can use ZooKeeper to help you realize it.

Features:

Eventual consistency: The data seen by the client is eventually consistent.
Reliability: The server saves the message, so it always exists.
Real-time: ZooKeeper cannot guarantee that two clients will get the newly updated data at the same time.
Independence (waiting is irrelevant): Different clients do not directly affect each other.
Atomicity: the update either succeeds or fails, there is no third state.

Note: To answer interview questions, do not just answer in one sentence. You can describe your understanding of the concept, characteristics and other aspects. Let me tell you, your purpose is to make the interviewer think you are good at communicating. Of course, if you don't know it, don't pretend to know it and say too many unprofessional ideas.

2. What are the application scenarios of ZooKeeper?

Data publishing and subscription

Publishing and subscribing is the so-called configuration management. As the name implies, it is to publish data to ZooKeeper nodes for subscribers to dynamically obtain data and realize centralized management and dynamic update of configuration information. For example, global configuration information, address lists, etc. are very suitable for use.

A common scenario of data publishing/subscribing is the configuration center, where the publisher publishes data to one or a series of nodes of ZooKeeper for subscribers to subscribe to and achieve the purpose of dynamically obtaining data.

Configuration information generally has several characteristics:

1. KV with small amount of data

2. Data content will change dynamically at runtime

3. Cluster machine sharing, consistent configuration

ZooKeeper uses a combination of push and pull.

1. Push: The server will push the Wathcer event notification to the client registered with the monitoring node

2. Pull: After the client gets the notification, it will actively go to the server to pull the latest data

naming service

As a distributed naming service, a naming service refers to obtaining the address of a resource or service through a specified name, using ZooKeeper to create a global path, which can be used as a name, pointing to the cluster in the cluster, the address of the service provided, Or a remote object etc.

The naming structure diagram of the unified naming service is as follows:

1. In a distributed environment, it is often necessary to uniformly name applications/services to facilitate identification of different services.

Similar to the correspondence between domain names and IPs, IPs are not easy to remember, but domain names are easy to remember.
Get the address, provider and other information of resources or services by name.

2. Organize service/application names in a hierarchical structure.

The service name and address information can be written to ZooKeeper, and the client can obtain the list of available services through ZooKeeper.

configuration management

The program is distributed and deployed on different machines, and the configuration information of the program is placed under the znode of ZooKeeper. When the configuration changes, that is, when the znode changes, you can change the content of a directory node in zk to use The watch notifies each client to change the configuration.

The ZooKeeper configuration management structure diagram is as follows:

1. In a distributed environment, configuration file management and synchronization is a common problem.

In a cluster, the configuration information of all nodes is consistent, such as Hadoop cluster.
After the configuration file is modified, it is hoped that it can be quickly synchronized to each node.

2. Configuration management can be implemented by ZooKeeper.

Configuration information can be written to a Znode on ZooKeeper.
Each node listens to this Znode.
Once the data in Znode is modified, ZooKeeper will notify each node.

cluster management

The so-called cluster management is: whether there are machines exiting and joining, and electing the master.

Cluster management mainly refers to cluster monitoring and cluster control. The former focuses on the collection of the running state of the cluster, while the latter focuses on the operation and control of the cluster. In development and operation and maintenance, in the face of clusters, there are often the following requirements:

1. Want to know how many machines are working in the cluster

2. Collect data on the runtime state of each machine in the cluster

3. Perform online and offline operations on the machines in the cluster

The cluster management structure diagram is as follows:

1. In a distributed environment, it is necessary to know the status of each node in real time, and some adjustments can be made according to the real-time status of the nodes.

2. It can be implemented by ZooKeeper.

Node information can be written to a Znode on ZooKeeper.
Listen to this Znode to get its real-time status changes.

3. Typical applications

Master status monitoring and election in Hbase.

Using the strong consistency of ZooKeeper can guarantee the global uniqueness of node creation in the case of distributed high concurrency, that is, if multiple clients request to create /currentMaster nodes at the same time, only one client request can be successfully created in the end

Distributed notification and coordination

1. In a distributed environment, there is often a service that needs to know the status of the sub-services it manages.

a) NameNode needs to know the status of each Datanode.

b) JobTracker needs to know the status of each TaskTracker.

2. The heartbeat detection mechanism can be realized through ZooKeeper.

3. Information push can be realized by ZooKeeper, which is equivalent to a publish/subscribe system.

distributed lock

Different services on different nodes may need to access some resources sequentially, and a distributed lock is needed here.

Distributed locks have the following characteristics: write locks, read locks, and timing locks.

Write lock: a temporary unnumbered node created on zk. Because it is an unordered number, it will not be automatically numbered when it is created, so only one client can get the lock and then write.

Read lock: Create a temporary numbered node on zk, so that even if the same node is created at the same time when a client joins next time, it will be automatically numbered, and the lock object can be obtained and then read.

Timing lock: A temporary numbered node created on zk controls the lock according to the size of the number.

distributed queue

There are two types of distributed queues:

1. When the members of a queue are all gathered, the queue is available, otherwise it has been waiting for all members to arrive. This is a synchronous queue.

a) A job consists of multiple tasks, and the job will run to completion only after all tasks are completed.

b) A /job directory can be created for the job, and then a temporary Znode is created for each completed task under this directory. Once the number of temporary nodes reaches the total number of tasks, it indicates that the job is completed.

2. The queue performs enqueue and dequeue operations according to the FIFO method, such as implementing the producer and consumer models.

3. Tell me about the working principle of Zookeeper?

The core of Zookeeper is atomic broadcast, which ensures the synchronization between servers. The protocol that implements this mechanism is called the Zab protocol.

The Zab protocol has two modes, which are recovery mode (master election) and broadcast mode (synchronization).

The full name of the Zab protocol is Zookeeper Atomic Broadcast** (Zookeeper Atomic Broadcast). Zookeeper uses the Zab protocol to ensure the final consistency of distributed transactions. The Zab protocol requires each Leader to go through three stages: discovery, synchronization, and broadcast.

When the service starts or after the leader crashes, Zab enters the recovery mode. When the leader is elected and most servers complete the state synchronization with the leader, the recovery mode ends. State synchronization ensures that the leader and server have the same system state.

In order to ensure the sequential consistency of transactions, zookeeper uses an incremental transaction id number (zxid) to identify transactions. All proposals have zxid added when they are proposed. In the implementation, zxid is a 64-bit number, and its upper 32 bits are used by epoch to identify whether the leader relationship has changed. Every time a leader is elected, it will have a new epoch, which identifies the current ruling period of that leader. The lower 32 bits are used to count up.

Epoch: It can be understood as the emperor's year name. When a new emperor leader is produced, there will be a new epoch year name.

Each Server has three states during the working process:

LOOKING: The current Server does not know who the leader is and is searching.
LEADING: The current Server is the elected leader.
FOLLOWING: The leader has been elected, and the current Server is synchronized with it.

4. Please describe Zookeeper's notification mechanism?

Zookeeper allows the client to register a Watcher with a znode on the server. When some specified events on the server trigger the Watcher, the server will send an event notification to the specified client to implement the distributed notification function, and then the client will follow the Watcher Notify state and event types to make business changes.

It is roughly divided into three steps:

Client registers Watcher

1. Call the three APIs getData, getChildren and exist, and pass in the Watcher object. 2. Mark the request and encapsulate Watcher to WatchRegistration. 3. Encapsulate it into a Packet object, and send the request to the server. 4. After receiving the response from the server, register the Watcher to ZKWatcherManager for management. 5. Request to return and complete the registration.

Server processing Watcher

1. The server receives and stores Watcher. 2. Trigger the Watcher 3. Call the process method to trigger the Watcher.

Client callback Watcher

1. The client SendThread thread receives the event notification, and the EventThread thread calls back the Watcher. 2. The Watcher mechanism of the client is also one-time, once triggered, the Watcher will be invalid.

The client will create a watcher event for a certain znode. When the znode changes, these clients will receive the zk notification, and then the client can make business changes according to the change of the znode.

5. Is Zookeeper's watch notification to the node permanent?

No, one-off . Whether it is a server or a client, once a Watcher is triggered, Zookeeper will remove it from the corresponding storage. This design effectively reduces the pressure on the server. Otherwise, for nodes that are updated very frequently, the server will continuously send event notifications to the client, which puts a lot of pressure on both the network and the server.

6. What are the roles in the Zookeeper cluster?

In a cluster, at least 3 are required. Or guarantee 2N + 1 units, which is an odd number. Why are odd numbers guaranteed? Mainly for the election algorithm.

7. What are the working statuses of the servers in the Zookeeper cluster?

LOOKING

Look for the Leader state; when the server is in this state, it will think that there is no Leader in the current cluster, so it needs to enter the Leader election state

FOLLOWING

Follower status; indicates that the current server role is Follower

LEADING

Leader status; indicates that the current server role is Leader

OBSERVING

Observer status; indicates that the current server role is Observer

8. How is the leader elected in the Zookeeper cluster?

When the Leader crashes, or loses most of the Followers, Zookeeper enters the recovery mode. The recovery mode needs to re-elect a new Leader, so that all the Servers return to a state of LOOKING.

Zookeeper has two election algorithms: based on basic paxos and based on fast paxos. The default is fast paxos. Due to space issues, here is the recommendation: [Distributed] Zookeeper's Leader election

9. How does Zookeeper ensure the sequential consistency of transactions?

Zookeeper uses an incremental transaction id to identify, and all proposals (proposals) are added with zxid when they are proposed. zxid is actually a 64-bit number.

The upper 32 bits are used by the epoch to identify whether the Leader has changed. If a new Leader is generated, the epoch will be incremented. The lower 32 bits are used to count up. When a new proposal is generated, it will first send a transaction execution request to other servers according to the two-stage process of the database. If more than half of the machines can execute and succeed, then the execution will start.

10. How do servers communicate with each other in the ZooKeeper cluster?

The Leader server will establish a TCP connection with each Follower/Observer server, and create an entity called LearnerHandler for each Follower/Observer.

LearnerHandler is mainly responsible for the network communication between Leader and Follower/Observer, including data synchronization, request forwarding and proposal voting.
The Leader server saves all Follower/Observer LearnerHandlers.

11. How is the ZooKeeper distributed lock implemented?

If there are N clients such as client 1 and client 2 competing for a Zookeeper distributed lock. Roughly as follows:

1. Everyone comes up and directly creates temporary ordered nodes one after another under a lock node

2. If you are not the first node, add a listener to your previous node

3. As long as the previous node releases the lock, it will be queued to the front, which is equivalent to a queuing mechanism.

And another purpose of using temporary sequential nodes is that it doesn’t matter if a client accidentally crashes itself after creating a temporary sequential node. Zookeeper will automatically delete the corresponding temporary sequential node when it senses that the client is down. It is used to automatically release the lock, or to automatically cancel its own queue.

Local locks can be implemented with JDK, but distributed locks must use distributed components. Such as ZooKeeper, Redis. There is a large section of online code, and interviews are generally not written. Let me talk about some key points.

A few points to note are as follows.

Deadlock problem: The lock cannot become a deadlock due to an accident, so a temporary node of ZK is used. If the client connection fails, the lock will be released automatically.
Lock waiting problem: locks have queuing requirements, so the sequential nodes of ZK are required.
Lock management problem: When a user releases the lock, other users need to be notified, so monitoring is required.
Herd effect of monitoring: For example, if there are 1,000 lock competitors and the lock is released, 1,000 competitors will be notified, and then judged, the one with the smallest serial number finally gets the lock. The other 999 competitors re-register to listen. This is the herd effect. If something happens, the whole flock will be alarmed. Each competitor should only monitor the node in front of him. For example, if No. 2 releases the lock, only No. 3 is notified.

12. Do you understand the system architecture of Zookeeper?

The main things we need to understand and master in the ZooKeeper architecture diagram are:

(1) ZooKeeper is divided into server (Server) and client (Client), and the client can connect to any server of the entire ZooKeeper service (unless the leaderServes parameter is explicitly set, the leader is not allowed to accept client connections).

(2) The client uses and maintains a TCP connection through which it sends requests, receives responses, obtains observed events, and sends heartbeats. If this TCP connection breaks, the client will automatically try to connect to another ZooKeeper server. When a client connects to the ZooKeeper service for the first time, the ZooKeeper server that accepts the connection establishes a session for the client. When the client connects to another server, the session will be re-established by the new server.

(3) Each Server in the above figure represents a machine on which the Zookeeper service is installed, that is, the entire cluster (or a pseudo-cluster) that provides the Zookeeper service;

(4) The servers that make up the ZooKeeper service must know each other. They maintain an in-memory state image, as well as transaction logs and snapshots in persistent storage, and the ZooKeeper service is available as long as a majority of servers are available;

(5) When ZooKeeper starts, it will elect a leader from the instance, and the leader is responsible for processing data updates and other operations. A successful update operation is marked if and only if most of the servers successfully modify the data in memory. Each server stores a copy of data in memory.

(6) Zookeeper can be replicated in clusters, and data consistency is maintained between clusters through the Zab protocol (Zookeeper Atomic Broadcast);

(7) The Zab protocol consists of two phases: the leader election phase and the Atomic Brodcast phase.

a) A leader will be elected in the cluster, and other machines are called followers. All write operations are sent to the leader, and all updates are told to the follower through brodcast.
b) When the leader crashes or the leader loses most of the followers, a new leader needs to be re-elected to restore all servers to a correct state.
c) When the leader is elected and most servers complete the state synchronization with the leader, the process of leader election is over, and it will enter the process of Atomic brodcast.
d) Atomic Brodcast synchronizes the information between the leader and the follower to ensure that the leader and the follower have the same system status.

13. Why is Zookeeper designed like this?

The purpose of ZooKeeper design is to provide high-performance, high-availability, sequentially consistent distributed coordination services, and to ensure the final consistency of data.

High performance (simple data model)

1. Organize data nodes in a tree structure;

2. All data nodes are stored in memory;

3. Follower and Observer directly process non-transactional requests;

High availability (building clusters)

1. If more than half of the machines survive, the service can run normally

2. Automatic Leader election

Sequential consistency (order of transaction operations)

1. Each transaction request will be forwarded to the Leader for processing

2. Each transaction will be assigned a globally unique incremental id (zxid, 64-bit: epoch + auto-increment id)

eventual consistency

1. Ensure the reliability of transaction submission by proposing voting

2. The proposed voting method can only ensure that more than half of the nodes can see the latest data after the client receives the successful submission of the transaction

14. Do you know what roles are in Zookeeper?

System model:

leader

The Leader server provides read and write services for clients. Responsible for the initiation and resolution of voting, and updating the system status.

learner

Follower (follower) The Follower server provides read services for clients, participates in the Leader election process, and participates in the write operation "more than half of the writes are successful" strategy.
Observer (observer) The Observer server provides read services for the client, does not participate in the Leader election process, and does not participate in the write operation "more than half of the writes are successful" strategy. It is used to improve the read performance of the cluster without affecting the write performance.

client

Service Request Initiator.

15. Are you familiar with Zookeeper node ZNode and related attributes?

What types of nodes are there?

There are two types of Znodes:

Persistent: After the client and server are disconnected, the created node will not be deleted (default).

Ephemeral: After the client and server are disconnected, the created node is deleted by itself.

Znode has four forms

Persistent directory node (PERSISTENT): After the client disconnects from Zookeeper, the node still has a persistent sequential number directory node (PERSISTENT_SEQUENTIAL)
After the client disconnects from Zookeeper, the node still exists, but Zookeeper sequentially numbers the node name: temporary directory node (EPHEMERAL)
After the client disconnects from Zookeeper, the node is deleted: Temporary Sequential Number Directory Node (EPHEMERAL_SEQUENTIAL)
After the client disconnects from Zookeeper, the node is deleted, but Zookeeper sequentially numbers the node name

"Note" : Set the sequence identifier when creating a ZNode, and a value will be appended after the ZNode name. The sequence number is a monotonically increasing counter maintained by the parent node.

What are the node attributes

A znode node can not only store data, but also have some other special properties. Next, we create a /test node to analyze the meaning of its various attributes.

[zk: localhost:2181(CONNECTED) 6] get /test

456

cZxid = 0x59ac //

ctime = Mon Mar 30 15:20:08 CST 2020

mZxid = 0x59ad

mtime = Mon Mar 30 15:22:25 CST 2020

pZxid = 0x59ac

cversion = 0

dataVersion = 2

aclVersion = 0

ephemeralOwner = 0x0

dataLength = 3

numChildren = 0

property description

16. Please briefly describe Zookeeper’s master selection process

The core of Zookeeper is atomic broadcast, which ensures the synchronization between servers. The protocol that implements this mechanism is called the Zab protocol. The Zab protocol has two modes, which are recovery mode (master election) and broadcast mode (synchronization). When the service starts or after the leader crashes, Zab enters the recovery mode. When the leader is elected and most of the servers complete the state synchronization with the leader, the recovery mode ends. State synchronization ensures that the leader and server have the same system state. Leader election is the key to ensuring the consistency of distributed data.

There are two main scenarios for elections: initialization and unavailable leader.

When one of the following two situations occurs on a server in the zk cluster, the leader election will start.

(1) The server is initialized and started.

(2) The connection with the leader cannot be maintained during the running of the server.

When a machine enters the leader election process, the current cluster may also be in the following two states.

(1) There is already a leader in the cluster.

(2) There is indeed no leader in the cluster.

First of all, in the first case, usually a certain machine in the cluster starts relatively late, and before it starts, the cluster has been working normally, that is, there is already a leader server. When the machine tries to elect a leader, it will be informed of the current server's leader information. It only needs to establish a connection with the leader machine and perform state synchronization.

The key point is that the leader is unavailable, and the leader selection system at this time.

Voting information contains two basic information.

sid: the server id, which is used to identify the serial number of the machine in the cluster.

zxid: the zookeeper transaction id number.

Every change of ZooKeeper state corresponds to an incremented Transaction id, which is called zxid. Due to the incremental nature of zxid, if zxid1 is smaller than zxid2, then zxid1 must occur before zxid2. Creating any node, or updating the data of any node, or deleting any node will cause the state of Zookeeper to change, resulting in an increase in the value of zxid.

A voting information is identified in the form of (sid, zxid).

For example: if the current server wants to recommend a server with sid of 1 and zxid of 8 to become the leader, then the voting information can be expressed as (1, 8)

After each machine in the cluster sends its own vote, it also accepts votes from other machines in the cluster. Each machine will process the votes received from other machines according to certain rules, so as to decide whether to change its own vote.

The rules are as follows:

(1) In the initial stage, everyone will vote for themselves.

(2) When receiving votes from other servers, it is necessary to compare other people's votes with your own votes. The rules are as follows: check zxid first. The server with a larger zxid is given priority as the leader. If the zxid is the same, the sid is compared, and the server with the larger sid is the leader.

The election process when all services start:

Three servers server1, server2, server3:

1. server1 starts, a machine will not be elected.

2. Start server2, change the status of server1 and server2 to looking, broadcast voting

3. Server3 starts, the state changes to looking, and joins broadcast voting.

4. In the state of first-time acquaintance, everyone does not know each other. Everyone thinks that they are the king, and they vote for themselves as the leader.

5. Instructions for voting information. The voting information is originally a five-tuple. Here, for clarity of logic, the expression is simplified.

For the first time, zxid = 0, and sid is the name of each node. This sid is configured in zoo.cfg and will not be repeated.

1. Initial zxid=0, server1 votes (1, 0), server2 votes (2, 0), server3 votes (3, 0)

2. When server1 receives the vote (2, 0), it will first verify the legitimacy of the vote, and then pk its own vote. The logic of pk is to compare zxid first, server1 (zxid) = server2 (zxid) = 0, zxid is equal Then compare sid, server1 (sid) < server2 (sid), pk, the result is that the vote of server2 wins. server1 updates its vote to (2, 0), and server1 votes again.

3. TODO Whether it is 2 or 3 in the end here needs to be determined through experiments.

4. When server2 receives the vote from server1, it will verify the legitimacy of the vote first, and then pk, its own vote wins, server does not need to update its own vote, after pk, it will resend the vote again.

5. Count the votes. After pk, the votes will be counted. If more than half of the nodes cast the same vote, it is determined that the Leader has been elected.

6. After the election is over, the status of the selected node changes from LOOKING to LEADING, and the status of other nodes participating in the election changes from LOOKING to FOLLOWING. If there is an Observer node, if the Observer does not participate in the election, its state is always OBSERVING before and after the election, without any change.

general speaking

Start voting -> node status becomes LOOKING -> each node chooses itself -> receives votes for PK -> big sid wins -> updates votes -> votes again -> counts votes, more than half of the votes results -> node The status is updated to the status of its own character.

17. Why is the number of Zookeeper clusters generally odd?

First of all, it is necessary to clarify the rules of zookeeper election: leader election requires the number of available nodes > total number of nodes/2.

For example: Marking whether a write is successful is considered valid only when more than half of the nodes send the write request successfully. Similarly, Zookeeper's choice of leader node is only valid when more than half of the nodes agree. Finally, whether Zookeeper is normal depends on whether more than half of the nodes are normal. This is based on the consistency principle of CAP.

Zookeeper has such a feature: as long as more than half of the machines in the cluster are working normally, the entire cluster is available to the outside world.

That is to say, if there are 2 zookeepers, as long as 1 zookeeper is dead, the zookeeper cannot be used, because 1 is not more than half, so the death tolerance of 2 zookeepers is 0;

Similarly, if there are 3 zookeepers, one of them is dead, and there are 2 normal ones left, more than half, so the tolerance of 3 zookeepers is 1;

In the same way:

2->0; two zookeepers, up to 0 zookeepers can be unavailable.
3->1; three zookeepers, at most one zookeeper may be unavailable.
4->1; four zookeepers, at most one zookeeper may be unavailable.
5->2; five zookeepers, up to 2 zookeepers may be unavailable.
6->2; two zookeepers, at most 0 zookeepers can be unavailable.
....

You will find a rule that the tolerance of 2n and 2n-1 is the same, both are n-1, so in order to be more efficient, why add that unnecessary zookeeper.

Zookeeper's election strategy also requires more than half of the nodes to agree to be the leader. If there are even nodes, it may lead to the same number of votes.

18. Do you know the principle of the Zookeeper listener?

1. Create a Main() thread.

2. Create two threads in the Main() thread, one is responsible for network connection communication (connect), and the other is responsible for listening (listener).

3. Send the registered listening event to Zookeeper through the connect thread.

4. Add the registered listening event to Zookeeper's registered listener list.

5. When Zookeeper detects that there is data or path changes, it sends this message to the Listener thread.

6. The process() method is called inside the Listener thread

19. Talk about the ACL permission control mechanism in Zookeeper

UGO（User/Group/Others）

It is currently used in the Linux/Unix file system and is also the most widely used permission control method. It is a coarse-grained file system permission control mode.

ACL (Access Control List) access control list

Including three aspects:

Permission Mode (Scheme)

(1) IP: Permission control from IP address granularity

(2) Digest: the most commonly used, using a permission identifier similar to username:password for permission configuration, which is convenient for distinguishing different applications for permission control

(3) World: The most open permission control method, which is a special digest mode with only one permission identifier "world:anyone"

(4) Super: super user

authorized object

Authorization object refers to the user to whom the authorization is granted or a specified entity, such as an IP address or a machine light.

Permission

(1) CREATE: Data node creation authority, allowing authorized objects to create child nodes under this Znode

(2) DELETE: child node deletion permission, allowing the authorized object to delete the child nodes of the data node

(3) READ: The read permission of the data node, allowing the authorized object to access the data node and read its data content or child node list, etc.

(4) WRITE: Data node update authority, allowing the authorized object to update the data node

(5) ADMIN: Data node management authority, allowing authorized objects to perform ACL related setting operations on the data node

20. What are the deployment modes of Zookeeper?

Zookeeper has three deployment modes:

1. Stand-alone deployment: run on a cluster;

2. Cluster deployment: multiple clusters running;

3. Pseudo-cluster deployment: A cluster starts multiple Zookeeper instances to run.

21. Does the Zookeeper cluster support dynamic addition of machines?

In fact, it is horizontal expansion, and Zookeeper is not very good in this regard. Two ways:

Restart all: Close all Zookeeper services and start them after modifying the configuration. Previous client sessions are not affected.

Restart one by one: Under the principle that more than half of the machines are alive and available, restarting a machine does not affect the external services provided by the entire cluster. This is the more common way.

Version 3.5 began to support dynamic expansion.

22. Describe the ZAB protocol

The ZAB protocol is a protocol defined by ZooKeeper itself, and its full name is ZooKeeper atomic broadcast protocol.

The ZAB protocol has two modes: how to recover when the Leader node crashes and how to broadcast messages to all nodes.

When the entire ZooKeeper cluster does not have a Leader node, it is a crash. For example, the cluster startup has just started, and the nodes do not know each other at this time. For example, the operation of the Leader node is down, or there is a network problem, and other nodes cannot ping the Leader node. At this time, the node collapse protocol in ZAB is required, and all nodes enter the election mode to elect a new Leader. The entire election process is realized by broadcasting. After the election is successful, everything needs to be based on the Leader's data, so data synchronization is required.

23. What is the connection and difference between ZAB and Paxos algorithm?

Same point:

(1) Both have a role similar to the Leader process, which is responsible for coordinating the operation of multiple Follower processes

(2) The Leader process will wait for more than half of the Followers to give correct feedback before submitting a proposal

(3) In the ZAB protocol, each Proposal contains an epoch value to represent the current Leader cycle, and the name in Paxos is Ballot

difference:

ZAB is used to build a highly available distributed data master and backup system (Zookeeper), and Paxos is used to build a distributed consistent state machine system.

24. How to deal with ZooKeeper downtime?

ZooKeeper itself is also a cluster, and it is recommended to configure an odd number of servers. Because of downtime, elections are required, and the election needs half of +1 votes to pass, in order to avoid a tie. Come in without an even number of servers.

If the Follower is down, it doesn't matter and doesn't affect any use. Users are unaware. If the leader goes down, the cluster has to stop external services and start elections. After a leader node is elected, data synchronization is performed to ensure that the data of all nodes and the leader are unified, and then external services begin.

Why do votes require half + 1? If half is enough, network problems may cause the cluster to elect two leaders, each with half of the younger brothers supporting it, so the data will be messed up.

25. Describe the idea of session management in ZooKeeper?

Bucket strategy:

Simply put, different session expirations may have time intervals, such as 15 seconds, 15.1 seconds, and 15.8 seconds. ZooKeeper uniformly expires these sessions in 16 seconds. This is very convenient for management. See the formula below. The expiration time is always an integer multiple of ExpirationInterval.

Calculation formula:

ExpirationTime = currentTime + sessionTimeout

ExpirationTime = (ExpirationTime / ExpirationInrerval + 1) * ExpirationInterval ,

see picture:

The default configured session timeout is 2tickTime~20tickTime.

26. What is the difference between ZooKeeper load balancing and Nginx load balancing?

ZooKeeper：

There is no single point problem, and the zab mechanism ensures that a single point of failure can re-elect a Leader
Only responsible for the registration and discovery of services, not for forwarding, reducing a data exchange (direct communication between the consumer and the server)
You need to implement the corresponding load balancing algorithm yourself

Nginx：

There is a single point problem, the single point load is high and the data volume is large, and it needs to be assisted by KeepAlived to achieve high availability
Each load acts as an intermediary forwarding role, which itself is a reverse proxy server
Built-in load balancing algorithm

27. Talk about the serialization of ZooKeeper

Serialization:

The memory data needs to be serialized when saving to the hard disk.
Memory data, transmitted to other nodes through the network, needs to be serialized.

The serialization protocol used by ZK is Jute, which provides the Record interface. The interface provides two methods:

serialize serialization method
deserialize deserialization method

The method to be serialized can be stored in the stream object in these two methods.

28. What is Zxid in Zookeeper and what does it do?

Zxid, that is, the transaction id, in order to ensure the sequential consistency of the transaction, ZooKeeper uses the incremental transaction Zxid to identify the transaction. Zxid will be added to the proposal. Zxid is a 64-bit number, and its upper 32 bits are used by Epoch to identify dynasty changes. For example, Epoch will be changed every time an election is made. The lower 32 bits are used for counting up.

Epoch: It can be understood as the age or cycle of the current cluster. Each Leader is like an emperor and has its own year name. Therefore, every time the dynasty changes, the Leader will add 1 to the previous era after the change. In this way, even after the old Leader crashes and recovers, no one will listen to it, because the Follower only obeys the orders of the current leader.

29. Explain the persistence mechanism of ZooKeeper

What is persistence?

Data, stored on disk or in a file.
After the machine restarts, the data will not be lost. The mapping of memory -> disk is somewhat similar to serialization.

Persistence of ZooKeeper:

SnapShot snapshot, recording the full amount of data in memory
TxnLog Incremental transaction log, recording each addition, deletion and modification record (checking is not a transaction log, and will not cause data changes)

Why is persistence such a hassle, is one not available?

The disadvantage of the snapshot is that the file is too large, and the snapshot file will not be the latest data. The disadvantage of incremental transaction log is that it takes a long time to run, there are too many logs, and the loading is too slow. A combination of the two works best.

Snapshot mode:

The data stored in the ZooKeeper memory in the DataTree data structure is periodically stored to the disk.
Since the snapshot file is a regular full backup of data, the data in the snapshot file is usually not the latest.

see picture:

30. What is the five-tuple of voting information in Zookeeper election?

Leader: SID of the elected Leader
Zxid: the transaction ID of the elected Leader
Sid: SID of the current server
electionEpoch: the current round of voting
peerEpoch: Epoch of the current server

Epoch > Zxid > Sid

Both Epoch and Zxid may be the same, but Sid must be different, so the two votes will definitely result in PK.

31. Talk about the split brain in Zookeeper?

To put it simply, Split-Brain means that when there are two nodes in your cluster, they both know that a master needs to be elected in this cluster. Then when there is no problem with the communication between the two of them, a consensus will be reached and one of them will be selected as the master. But if there is a problem with the communication between them, the two nodes will feel that there is no master now, so each elects itself as the master, so there will be two masters in the cluster.

For Zookeeper, there is a very important question, that is, what kind of situation is used to judge whether a node is dead or down? In a distributed system, these are judged by monitors, but it is also difficult for monitors to determine the status of other nodes. The only reliable way is heartbeat. Zookeeper also uses heartbeat to determine whether the client is still alive.

Using ZooKeeper to do Leader HA is basically the same way: each node tries to register a temporary node that symbolizes the leader, and others that fail to register become followers, and monitor the temporary nodes created by the leader through the watch mechanism. Zookeeper passes The internal heartbeat mechanism is used to determine the status of the leader. Once an accident occurs to the leader, Zookeeper can quickly learn about it and notify other followers, and other flowers will respond accordingly, thus completing a switch. Part of this is done. But there is a very serious problem here. If you don’t notice it, it will cause a split brain in the system in a short period of time. Because the heartbeat timeout may be the leader hangs up, but it may also be a problem with the network between the zookeeper nodes, resulting in the leader In the case of suspended animation, the leader did not actually die, but a problem with the network with ZooKeeper caused Zookeeper to think it was down and then notified other nodes to switch, so that one of the followers became the leader, but the original leader did not At this time, the client also gets the message of leader switching, but there will still be some delays. Zookeeper needs to communicate and needs to be notified one by one. At this time, the whole system is very chaotic. Maybe some clients have been notified to connect to the new leader. , Some clients are still connected to the old leader. If two clients need to update the same data of the leader at the same time, and these two clients are connected to the old and new leaders at the moment, serious problems will occur.

Here is a brief summary: Feign death: The leader is thought to be dead due to heartbeat timeout (caused by network reasons), but in fact the leader is still alive. Split brain: Due to suspended animation, a new leader election will be initiated to elect a new leader, but the old leader network is connected again, resulting in two leaders. Some clients connect to the old leader, and some clients connect to the old leader. Then connect to the new leader.

32. What causes Zookeeper split brain?

The main reason is that the Zookeeper cluster and the Zookeeper client cannot be completely synchronized when judging the timeout. At the same time, there is also a sequence of speeds for notifying each client after discovery and switching. Generally, the probability of this situation is very small, and the leader node needs to be disconnected from the Zookeeper cluster network, but there is no problem with the network between other cluster roles, and the above conditions must be met, but once it occurs, it will cause serious consequences. The data is inconsistent.

33. How does Zookeeper solve the split-brain problem?

To solve the Split-Brain problem, there are generally the following methods: Quorums (quorum) method: For example, in a cluster with 3 nodes, Quorums = 2, that is to say, the cluster can tolerate the failure of 1 node. One leader can be elected, and the cluster is still available. For example, for a cluster with 4 nodes, its Quorums = 3, and the Quorums must exceed 3, which means that the tolerance of the cluster is still 1. If two nodes fail, the entire cluster is still invalid. This is the default method used by zookeeper to prevent "split brain".

Redundant communications (redundant communication) mode is used: multiple communication modes are used in the cluster to prevent the nodes in the cluster from being unable to communicate due to the failure of one communication mode.

Fencing (shared resources) method: For example, if you can see the shared resources, it means that you are in the cluster. The leader who can obtain the lock of the shared resources is the leader. If you can’t see the shared resources, you are not in the cluster.

It is actually very simple to avoid zookeeper "brain split". When the follower node switches, it does not switch immediately after checking that there is a problem with the old leader node, but sleeps for a sufficient period of time to ensure that the old leader has been informed of the change and After doing the relevant shutdown cleanup work and then registering as the master, this kind of problem can be avoided. The sleep time is generally defined as the timeout period defined by zookeeper, but the system may not be available during this period, but It's worth it for the consequences of inconsistent data.

1. ZooKeeper adopts Quorums by default to prevent "split brain" phenomenon. That is, only more than half of the nodes in the cluster vote to elect a Leader. This way can ensure the uniqueness of the leader, either the only leader is elected, or the election fails. The role of Quorums in zookeeper is as follows:

The minimum number of nodes in the cluster is used to elect a leader to ensure that the cluster is available.
Inform clients that the data has been saved by the minimum number of nodes in the cluster before the data is safely saved. Once these nodes have saved the data, the client will be notified that it has been safely saved and can continue with other tasks. The remaining nodes in the cluster will eventually hold the data as well.

Assuming a leader dies in suspended animation, the rest of the followers elect a new leader. At this time, the old leader is revived and still considers itself to be the leader. At this time, it will also be rejected to send write requests to other followers. Because whenever a new leader is generated, an epoch label (identifying the current ruling period of the leader) will be generated. This epoch is incremental. If the followers confirm the existence of the new leader and know its epoch, they will reject the epoch being smaller than the current leader. All requests for epoch. Is there any follower who does not know the existence of the new leader? It is possible, but certainly not most of them, otherwise the new leader cannot be generated. Zookeeper's writing also follows the quorum mechanism. Therefore, writing that is not supported by the majority is invalid. Even if the old leader considers himself a leader, it still has no effect.

In addition to using the above default Quorums method to avoid "split brain", zookeeper can also take the following preventive measures: 2. Add redundant heartbeat lines, such as double line lines, to minimize the chance of "split brain". 3. Enable disk lock. The serving party locks the shared disk, and when a "split brain" occurs, the other party is completely "unable to take away" the shared disk resources. But using a lock disk also has a big problem. If the party occupying the shared disk does not actively "unlock", the other party will never get the shared disk. In reality, if the service node suddenly crashes or crashes, it is impossible to execute the unlock command. The backup node cannot take over shared resources and application services. So someone designed a "smart" lock in HA. That is, the serving party only enables the disk lock when it finds that the heartbeat lines are all disconnected (the peer end cannot be detected). It is usually not locked. 4. Set up an arbitration mechanism. For example, set a reference IP (such as gateway IP). When the heartbeat line is completely disconnected, both nodes will ping the reference IP. If it fails, it means that the breakpoint is at the local end, not only "heartbeat", but also external "service". If the local network link of the local end is broken, even if the application service is started (or continued), it is useless, so the competition is voluntarily given up, and the end that can ping the reference IP can start the service. To be safer, the party that cannot ping the reference IP simply restarts itself to completely release those shared resources that may still be occupied.

34. Tell me about the trade-offs made on the CAP issue of Zookeeper?

Consistency C : Zookeeper is a strong consistency system. In order to ensure strong availability, the data synchronization method of "more than half success means success" may cause data inconsistency on some nodes. So Zookeeper also provides the sync() operation to synchronize the data of all nodes, which leaves the choice of C and A to the user, because using sync() will inevitably prolong the synchronization time and cause some loss of availability.

Availability A : Zookeeper data is stored in memory, and each node can respond to read requests, with good response performance. Zookeeper guarantees that data is always available without locks. And more than half of the nodes have the latest data.

Partition tolerance P : Too many follower nodes will increase the delay of data synchronization (more than half of the followers need to finish writing and committing). At the same time, the convergence speed of the election process will be slower and the availability will be reduced. Zookeeper alleviates this problem by introducing observer nodes. After adding observer nodes, the cluster can accept more nodes for client requests, and observers do not participate in voting, which can improve availability and scalability, but multi-node data synchronization is always a problem, so consistency will decreased.

35. Why is watch monitoring one-off?

If the server changes frequently and there are many monitoring clients, all clients must be notified of each change, which puts a lot of pressure on the network and server.

Generally, the client executes getData(node A,true). If node A is changed or deleted, the client will get its watch event, but later node A changes again, and the client does not set the watch event. It is no longer sent to the client.

In practical applications, in many cases, our client does not need to know every change of the server, I only need the latest data.