Knowledge finishing, ZooKeeper face questions

EDITORIAL

Compiled some Internet giant face questions, questions often asked these surfaces, as Java engineer also need to have some knowledge, after all, the combination of theory and practice, is king, fragmentation finishing, nibbling some knowledge every day , happy every day, if you have help, remember that the point of a concern and a point like oh .

Related series

Knowledge finishing, MyBatis face questions
knowledge consolidation, ZooKeeper face questions

What ZooKeeper that?

ZooKeeper is an open source distributed coordination service, which is a cluster manager, monitors the status of each node in the cluster to the next step reasonable operating according to the feedback node submitted. Finally, the easy to use interface and efficient performance, function and stability of the system to the user.
Distributed applications, such as data released under Zookeeper realization / subscribe, load balancing, name services, distributed coordination / notification, cluster management, Master election, distributed locking and distributed queue functions.
Zookeeper distributed to ensure the consistency of the following characteristics:

• Sequential consistency
• Atomicity
• A single view
• reliability
• Real-time (eventual consistency)

Client read request may be processed in any machine in the cluster, if the read request to register the listener on the node, the listener is processed by a zookeeper connected machine. For written requests, which will be simultaneously sent to other machines and the zookeeper agreement, the request will return success. Therefore, with the increase zookeeper cluster machine, read request throughput increases but write request throughput will decrease.

Orderliness is a very important characteristic of a zookeeper, all updates are global ordered, each update has a unique time stamp, the time stamp called zxid(Zookeeper Transaction Id). But only with respect to the read request to update orderly, that is, the read request will return results with that zookeeper latest zxid.

ZooKeeper provides what?

• File system
• Notification mechanism

Zookeeper File System

Zookeeper provide a multi-level namespace node (called a node znode). And the file system is different is that these nodes can set the data associated with it, and the file system can only store data file node directory node can not. Zookeeper In order to ensure low latency and high throughput, maintained in memory of this tree directory structure, this feature can not be used to store Zookeeper such large amounts of data, each node storing data limit is 1M .

ZAB agreement?

• ZAB协议It is an atomic broadcast protocol support crash recovery for distributed coordination service designed Zookeeper.
• ZAB 协议The two basic modes: the broadcast message and crash recovery.

When the entire cluster zookeeper just started or Leader server is down, restart or network failure resulting in more than half of the server and the server Leader maintain normal communication does not exist, all processes (server) into the crash recovery mode, first elected a new Leader server and cluster in Follower server starts data with the new Leader server synchronization, when a cluster than half machine completes the data with the Leader server synchronization, exit recovery mode incoming messages broadcast mode, Leader server starts receiving the client's transaction request generating things proposals for transaction request processing.

Four types of data nodes Znode

• PERSISTENT-持久节点: Unless manually deleted, or node has been present on the Zookeeper
• EPHEMERAL-临时节点: Life Cycle temporary node with the client session binding, once the client session expires (the client and not necessarily zookeeper disconnected session expires), then the client creates all temporary nodes will be removed.
• PERSISTENT_SEQUENTIAL-持久顺序节点 : Basic characteristics with persistent nodes, only increased the order attribute, node name is appended behind an auto-incremented integer numbers maintained by the parent.
• EPHEMERAL_SEQUENTIAL-临时顺序节点 : The basic characteristics of the temporary node with an increase of the order of attributes, node name is appended behind an auto-incremented integer numbers maintained by the parent.

Zookeeper Watcher mechanism - data change notification

Zookeeper allows the client a Watcher to listen to a Znode registered service side, while some specified event server triggers the Watcher, the server sends an event notification to a specified client to achieve distributed notification, then the client according Watcher notification status and event types to make a change on the business.
Working Mechanism:

• Client registration watcher
• Server processing watcher
• Client callback watcher

Watcher Summary of Features:

• Disposable
  either the server or the client, once a Watcher is triggered, Zookeeper will be removed from the appropriate store. This design effectively reduces the pressure on the service side, or for heavily-updated node, the server will continue to send event notifications to the client, regardless of the network or server-side pressures are very large.
• Client serial execution
  process client callback Watcher is a serial process synchronization.
• Lightweight
  • Watcher notice is very simple, just tell the client event has occurred without the specific details of the event.
  • Watcher client registers to the server when the client does not put real object entity Watcher passed to the server, just use boolean type attribute tagged in the client request.
• watcher eventSend watcher asynchronous notification event sent from the server to the client is asynchronous, lies the problem, communication between different client and server through socket, due to network delays or other factors cause the client at the moment does not make sense to listen to the event Since Zookeeper itself provides ordering guarantee, namely the client to listen to events, will perceive it monitors znode changed. So we can not expect to be able to monitor the use of Zookeeper changes to each node. Zookeeper can only ensure that the final agreement, but can not guarantee strong consistency.
• registered watcher getData、exists、getChildren
• trigger watcher create、delete、setData
• When a client connects to a new server, watch the session will be triggered in any event. When a lost connection with the server is unable to receive the watch. And when the client reconnects, if necessary, all previously registered watch, will be re-registered. Usually this is completely transparent. Only in a special case, the watch may be lost: for a nonexistent one znodeof exist watch, if you disconnect the client during the connection is created, and then before the client connections and removed, in this case, this watch events may be lost.

Watcher achieve client registration

• Call getData()/getChildren()/exist()three API, the object passed Watcher
• Marker request request, encapsulating the Watcher WatchRegistration
• Packet encapsulated into objects, server sends request sent
• After receiving the server response, the Watcher registered to ZKWatcherManager for management
• Request to return to complete the registration.

Server process to achieve Watcher

• Server receives Watcher and stores the received client requests, whether to process the request determination requires registration Watcher, if desired representing node of the path data nodes and ServerCnxn (ServerCnxn connected to a client and server, to achieve the Watcher of process interfaces, then Watcher can be seen as an object) is stored to and in WatcherManager watch2Paths in the WatchTable.
• Watcher trigger service receives setData()a transaction request triggers NodeDataChanged incident as an example:
  • The package WatchedEvent notification state (SyncConnected), event type (NodeDataChanged) and packaged into a node path objects WatchedEvent
  • Find query Watcher Watcher node according to the path from WatchTable
  • Did not find; no explanation Watcher client registered on the data node
  • Find; extracted from WatchTable and Watch2Paths and delete the corresponding Watcher (From here we can see Watcher is a one-time service in the end, the trigger once ineffective)
• Call the process method to trigger this process Watcher is mainly through ServerCnxn corresponding TCP connection to send Watcher event notification.

Client callback Watcher

SendThread client thread receives event notifications, referred EventThread thread callback Watcher. Watcher mechanisms client is also a one-time, once triggered, the Watcher becomes ineffective.

ACL access control mechanisms

• UGO (User / Group / Others)
  currently used in Linux / Unix file systems, but also the most widely used access control mode. It is a coarse-grained file system access control mode.
• ACL (Access Control List) Access Control List
  includes three aspects:
  • Permissions mode (Scheme)
    • IP: IP address control permissions granularity
    • Digest: The most commonly used, in a manner similar username:passwordto the configuration of privilege authority identifiers, to facilitate application to distinguish between different access control
    • World: access control most open way, is a special digest mode, only one authorization ID“world:anyone”
    • Super: Super User
  • Authorization object
    authorization object refers to permissions granted to a user or a designated entity, e.g. the IP address or machine lights.
  • Permissions Permission
    • CREATE: Permission to create data node, the child node object is created that allows authorized under the Znode
    • DELETE: Child node delete permissions to allow authorization object to delete the data node's children
    • READ: Read permission data node, allowing authorized access to the data subject node and read its data content or child node lists, etc.
    • WRITE: Data node update permissions to allow authorized object to update the data node
    • ADMIN: Data node management authority that allows authorized object ACL-related operations on the data set node

Chroot properties

After version 3.2.0, adds Chroot feature that allows each client to set a namespace for themselves. If a client is set Chroot, then the client any operation on the server, it will be limited in its own namespace.

By setting Chroot, a client can be used in a sub-tree Zookeeper server correspond to those at the scene Zookeeper multiple applications into a utility group, to achieve mutual isolation between different applications is very helpful.

Session Management

• Sub-barrel strategy: the similar session in the same block management, so that a unified session isolation and treatment of different blocks of the same block in Zookeeper.
• Allocation principles: "The next point timeout" for each session (ExpirationTime)
• The formula is:

ExpirationTime_ = currentTime + sessionTimeout
ExpirationTime = (ExpirationTime_ / ExpirationInrerval + 1) * ExpirationInterval 
ExpirationInterval 是指 Zookeeper 会话超时检查时间 间隔，默认 tickTime

Server Role

• Leader
  • Scheduling and unique transaction request handlers, assure an orderly cluster transaction
  • Within a cluster each service dispatcher
• Follower
  • Non-transaction processing the client's request, forwarding the transaction request to the server Leader
  • Proposal of transaction requests to vote
  • Participation in elections Leader
• Observer
  • After the introduction of a version 3.0 server role, without affecting the cluster transaction processing capabilities of the non-transactional capabilities to enhance the cluster
  • Non-transaction processing the client's request, forwarding the transaction request to the server Leader
  • Voting does not participate in any form of

Zookeeper working condition at Server

Server has four states, namely, L OOKING、FOLLOWING、LEADING、OBSERVING.

• LOOKING: Finding Leader status. When the server is in this state, it does not think the current cluster Leader, Leader is required to enter the state election.
• FOLLOWING: Follower state. It indicates that the current server role is Follower.
• LEADING: Leader state. It indicates that the current server role is a Leader.
• OBSERVING: Observer status. It indicates that the current server role is Observer.

data synchronization

After the completion of the entire cluster Leader elections, Learner (Follower and Observer collectively) back to the server to register with the Leader. When the Leader Learner server wants to complete the registration server, enter data synchronization link. Data synchronization process are performed :( way messaging)

• Learner 向 learners 注册
• data synchronization
• Synchronization confirmation

Zookeeper data synchronization is usually divided into four categories:

• Direct synchronization difference (DIFF synchronous)
• First rollback then differentiated synchronization (TRUNC + DIFF synchronization)
• Only rollback synchronization (TRUNC synchronous)
• The full amount of synchronization (SNAP synchronous)

Performing data synchronization ago, Leader will complete server data synchronization initialization:

• peerLastZxid: ACKEPOCH message transmitted from the registration server when the learner extracts lastZxid (ZXID Learner server that last processed)
• minCommittedLog: Leader Proposal cache server queue committedLog smallest ZXID
• maxCommittedLog: Leader server Proposal buffer queue committedLog largest ZXID

Direct synchronization difference (DIFF synchronous)
scenarios: peerLastZxidbetween minCommittedLogand maxCommittedLogamong

First rollback and then differentiated synchronous (TRUNC + DIFF synchronous)
scenarios: When the server finds a new Leader Learner server contains a transaction record he did not, then you need to let the Learner server for transaction rollback - rollback to the Leader exist on the server, but also the closest to peerLastZxidtheZXID

Only rollback synchronization (sync TRUNC)
Scene: peerLastZxidgreater thanmaxCommittedLog

The full amount of synchronization (SNAP synchronous)

• Scene One: peerLastZxidless thanminCommittedLog
• Scene 2: Leader No Proposal on the server, and a buffer queue peerLastZxidis not equal tolastProcessZxid

How zookeeper is to ensure the consistency of the order of transactions?

zookeeper increasing use of global transaction Id identified, all of proposal (proposal) are proposed to be added when zxid, zxid is actually a 64-bit digital, 32-bit high is epoch (period; era; the world; new age) period leader used to identify if there is a new generation leader out, Epoch will be incremented, the lower 32 bits used to count up. When the newly generated proposal, the database will be based on two-stage process, first of all transaction requests are issued to the other server, if more than half of the machine can perform and be successful, then it will begin.

Distributed cluster Why Master?

In a distributed environment, some business logic needs only one of the machines in the cluster for execution, other machines can share the results, which can greatly reduce double counting, improve performance, so he needs to be leader election.

How to deal with zk node goes down?

Zookeeper itself is a cluster, the recommended configuration of not less than three servers. Zookeeper also must ensure that when a node goes down, the other nodes will continue to provide services. If a Follower is down, there are two servers to provide access to the data on the Zookeeper is because there are multiple copies of the data will not be lost; if it is a Leader down, Zookeeper will elect a new Leader. ZK cluster mechanism is normal as long as more than half of the nodes, the cluster will be able to provide normal services. Only in node ZK hang too much, only half or less than half of the nodes can work, only the cluster fail. Therefore, cluster 3 nodes can hang one node (2 votes obtained Leader> 1.5). cluster 2 nodes will not hang up on any one of the nodes (1 votes obtained Leader <= 1)

zookeeper nginx load balancing and load balancing difference

zk load balancing can be regulated, but nginx can adjust the weight, other needs are controllable need to write their own plug-ins; however nginx throughput is much larger than zk, it should be said by business choose which way to use.

Zookeeper, which has several several deployment models?

Deployment modes: single mode, pseudo-cluster model, the cluster model.

Cluster at least to several machines, clustering rules is what?

Cluster sets rules 2N + 1, N> 0, i.e. three.

Cluster support dynamically adding machine?

In fact, the level of expansion, Zookeeper in this area is not very good. In two ways:

• All Restart: Close all Zookeeper service, started after modifying the configuration. It does not affect the session prior to the client.
• Restart one by one: more than half under the principle of survival can be used, a machine reboot does not affect the entire cluster to provide services. This is a more common way.

Since version 3.5 supports dynamic expansion.

Zookeeper watch monitor notice of nodes is permanent? Why is not permanent?

No. Official statement: Watch a trigger event is a one-off, when set up Watch the changed data when the server will send the change to set up a client's Watch, in order to inform them.

Why is not permanent, for example, if the server changes frequently, and monitor clients in many cases, each change requires notification to all clients, caused great pressure on networks and servers. General client implementation of getData ( "/ node A", true), if node A has occurred to change or delete, the client will get its watch events, but after node A has undergone change, and the client has not set watch an event, do not give the client sends. In practical applications, in many cases, our client does not need to know every time the server changes, the latest data as long as I can.

Zookeeper What are the java client?

java client: zk comes zkclientand Apache open source Curator.

What is chubby, and zookeeper how to see than you?

chubby is google, the full realization paxos algorithm, not open source. zookeeper is chubby open source implementation, use zab protocol variants paxos algorithm.

Zookeeper say a few commonly used commands

Commonly used commands: ls get set create deleteand so on.

Contact with the distinction ZAB and Paxos algorithm?

Same point:

• Both are present role similar to Leader process, which is responsible for coordination of multiple processes running Follower
• Leader after the process will wait for more than half of Follower make the right feedback, will submit a proposal
• ZAB agreement, each of the Proposal contains an epoch value to represent the current Leader period, Paxos the name for Ballot

difference:

• ZAB standby system used to build highly available distributed data (Zookeeper), Paxos coherency state is used to build distributed systems.

Zookeeper typical application scenarios

Zookeeper distributed data management and coordination framework of a typical publish / subscribe model, developers can use it to publish and subscribe to distributed data.

Zookeeper by the wealth of data nodes cross-use, with the Watcher event notification mechanism, you can build a series of distributed core feature of middle-aged will involve very convenient, such as:

• Data Publish / Subscribe
• Load Balancing
• Naming Service
• Distributed Coordination / Notification
• Cluster Management
• Master election
• Distributed Lock
• Distributed Queue

Data Publish / Subscribe

• Description: Data publish / subscribe systems, so-called distribution center, as the name suggests is a publisher to publish data for subscriber data subscription.
• Dynamic updates acquired data (configuration information) for data (configuration information) and centralized management of data: object
• Design Mode: Push mode, Pull mode
• Data (configuration information) properties
  • The amount of data is usually small
  • Data content at runtime dynamic updates occur
  • Each cluster shared machine, configured with the same, such as: the machine list information, the runtime switch configuration database configuration information
• Based implementation of Zookeeper
  • Data storage: the data (configuration information) is stored on a data node Zookeeper
  • Data Acquisition: The Zookeeper start initialization data node node data read from and sign up for a data node on the change Watcher
  • Data Change: When the change data, node data corresponding to the update Zookeeper, Zookeeper data change notification will be sent to each client, the client after receiving the notice to re-read the data after the change.

Load Balancing

• zk naming services
  Naming Service is the address to access to resources or services by specifying the name, use zk create a global path, this path can be used as a name, pointing to a cluster of clusters, address of the service provided, or a remote objects, and so on.
• Distributed notification and coordination
  for the system for scheduling: the operator to send a notification via the console is actually changing the state of a node, then zk these changes will be sent to all registered clients watcher of this node. For the implementation of the report: Each worker process to create a temporary node in a directory. And carrying the work schedule data, such processes can monitor changes summary catalog of child nodes of the global real-time progress of the case.
• zk naming service (file system)
  naming service is the address to access to resources or services by specifying the name, use zk create a global path, that is the only path, the path can be used as a name, pointing to a cluster clusters, address and services, or a remote object, and so on.
• zk configuration management (file system notification mechanism)
  program distributed deployed on different machines, the configuration information on the program under znode zk when there are configuration changes, that is, when znode changes, you can zk change the content in a directory node using the watcher to notify each client in order to change the configuration.
• Zookeeper Cluster Management (file system notification mechanism)
  so-called cluster management not care about two things: Is there a machine quit and join the electoral master. On the first point, all the machines agreed to create a temporary directory node in the parent directory, and then change the message the child node listens parent directory node. Once the machine hang up, the machine is connected to the zookeeper's off, it creates a temporary directory node is removed, all other machines have notified: a sibling directory is removed, so everyone knows: it on board a. The new machine is similar to join, all machines receive notification: new sibling to join, highcount there has been, for the second point, we change a little bit, all the machines in order to create a temporary directory node number, select the smallest number each time the machine as a master just fine.
• Zookeeper distributed lock (file system notification mechanism)
  have zookeeper consistency of the file system, lock the problem easier. Lock service can be divided into two categories, one is to maintain exclusivity, the other is to control the timing. For the first category, we will znode on a zookeeper seen as a lock, which is achieved by createznode way. All clients to create /distribute_locknodes, that create the ultimate success of the client that is also owned the lock. Exhausted removed to create their own distribute_locknode to release the lock. For the second category, / distribute_lock already pre-existing, all clients to create a temporary directory node sequentially numbered under it, and choose the same master, the lowest-numbered lock was exhausted deleted, and convenient.
• Zookeeper queue management (file system notification mechanism)
  are two types of queues:
  • Synchronous queue, when they gathered all the members of a queue, the queue is available, otherwise it has been waiting for all members to arrive.
  • Queue enqueue and dequeue operation in FIFO mode.
    The first category, create a temporary directory node in the directory under the agreement, whether the number is the number of nodes monitor our requirements.
    The second type, and a distributed lock service control sequence consistent with the basic principles of the scene, the numbered columns, the columns by number. Created under the specific directory PERSISTENT_SEQUENTIALWatcher notification waiting queue node, create a successful, delete the queue smallest node number for consumption. Under this scenario the znode Zookeeper for storing message data is stored znode message content, the message queue of SEQUENTIALthe sequence number is the number of messages, can be removed sequentially. Since the node are creating a persistent, so do not worry about losing issue queue messages.