Zookeeper learning summary

cap : Consistent Availability Partition Fault Tolerance

BASE principle: basic availability + soft state (synchronization allows delay) + eventual consistency


Consistency protocol: 2PC (2-phase commit) + 3PC (3-phase commit)

3PC = canCommit + preCommit + doCommit

PAXOS introduces the principle of more than half, as long as more than half pass, then pass

zookeeper provides efficient and reliable distributed coordination services for distributed applications

zookeeper adopts zab protocol zookeeper atomic broadcast

zookeeper is a solution for distributed data consistency

Single view: The client sees the same view regardless of which server it connects to

zookeeper guarantees eventual consistency

zookeeper stores full data in memory to achieve high throughput and low latency


As long as half of the machines in the cluster serve external services, they can provide external services.

zk has three roles: leader + foller + observer Both foller and observer can provide external read services when only foller can elect


Znode is divided into temporary node and persistent node  

Persistent nodes need to be actively deleted to be deleted

Ephemeral nodes are related to the time the client connects to the server session The node is deleted once the client disconnects

Each znode will maintain a stat data structure to maintain these three versions: version (znode version) + cversion (znode child node version) + aversion (znode acl version)

watcher allows to register events of interest to clients on the node

acl=access control list access control set=CREATE READ WRITE DELETE ADMIN	

zookeeper mainly relies on zab for data consistency

Zab only allows one master process to accept all requests from all clients and publish transactions to replica processes using the zab protocol.


All transaction requests can only be converted into transactions by the leader server and distributed to all followers for more than half of the proposals to pass

The zab protocol is divided into crash recovery (electing a new leader) and message broadcasting

The leader server will assign a transaction id (zxid) to each transaction proposal

zxid consists of 64 bits. The low 32 bits are a simple monotonically increasing counter used to record the client's request id each time. The high 32 bits are used to store the epoch value +1 for each election.


The zab protocol is the core of zk. It stipulates that only one main process broadcasts messages at any time. After the main process fails, a new leader must be elected.

The zab protocol has three phases: discovery (looking election) + synchronization (last transaction) + broadcast (accepting the client's response)
 
The leader process communicates with the follower process through the heartbeat detection mechanism   

If the leader fails to obtain more than half of the followers' heartbeats within the timeout period, it will enter the looking state (re-election).

Each transaction proposal in the zab protocol has an epoch value representing the current leader cycle  




use zk
---------------------------------------
zoo.cfg configuration file
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit = 10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# Mirror data location
dataDir=D:\\zookeeper\\data\\zookeeper
#log location
dataLogDir=D:\\zookeeper\\logs\\zookeeper
#Client connection port
clientPort=2181
#Format in the cluster: server.id=ip:port:port
server.1=ip1:2888:3888
server.2=ip2:2888:3888
server.3=ip3:2888:3888

All zoo.cfg files in the cluster are consistent

The content of a myid file generated under dataDir is the value of id in zoo.cfg such as 1 and only one number


create -s sequential nodes -e temporary nodes are persistent nodes if they are not added by default

ls / to view

get path is used to get the node content

set path value is used to update node content

delete path is used to delete a node


Permission control: control access by setting acl permissions in the node


zookeeper.addAuthInfo("digest","username:password");// Similar username: password format
zookeeper.create(path,value,Ids.CREATOR_ALL_ACL,NodeType);

try {
	// establish a connection
	ZooKeeper zk = new ZooKeeper("127.0.0.1:2181", 5000, null);
	System.out.println(zk.getState());
	zk.addAuthInfo("digest", "lanwx:123".getBytes());
	//zk.create("/zk-lanwx/childss", "lan".getBytes(), Ids.CREATOR_ALL_ACL, CreateMode.PERSISTENT);
	// No permission
	ZooKeeper zk1 = new ZooKeeper("127.0.0.1:2181", 5000, null);
	System.out.println(new String(zk1.getData("/zk-lanwx/childss", null, null)));
} catch (Exception e) {
	e.printStackTrace ();
}




Curator works well as a zookeeper client API for distributed locks


zookeeper usage scenarios
-------------------------
zk is a highly available distributed data consistency framework

All machines in the cluster share consistent data


YARN's new generation of distributed scheduling framework


Technology Insider
-------------------------
Data Model: Tree Structure


Each zxid represents an update operation


znode node type: persistent + ephemeral + sequential

zookeeper uses jute to serialize and deserialize data

zookeeper adopts tcp/ip communication protocol

A session exists after the client establishes a connection with the server