Please, stop asking me how Zookeeper implements distributed locks!

Guide

• Where there are people (locks), there are rivers and lakes (affairs). Today, without talking about rivers and lakes, come and sultry people.
• The concept of distributed locks and why distributed locks are used must be clear to everyone. I wrote about how Redis implements distributed locks some time ago. Today's article talks about how Zookeeper implements distributed locks.
• Today, I will talk about how ZK implements distributed locks in detail from the following aspects: ZK's four types of exclusive locks, read and write locks, Curator, distributed lock

The four nodes of ZK

• Persistent node: the node will always exist after it is created
• Temporary node: The life cycle of the temporary node is bound to the current session. Once the current session is disconnected, the temporary node will also be deleted. Of course, it can be actively deleted.
• Persistent ordered nodes: Node creation has always existed, and zk will automatically add a self-increasing suffix to the node as the new node name.
• Temporary ordered nodes: retain the characteristics of temporary nodes, and zk will automatically add a self-increasing suffix to the node as the new node name.

Implementation of exclusive lock

• The implementation of exclusive locks is relatively simple, taking advantage of the feature that zk can't create duplicate nodes . As shown below:

 

• According to the analysis in the above figure, it is roughly divided into the following steps: try to acquire the lock: create a temporary node, zk will ensure that only one client is successfully created. The creation of the temporary node is successful, the lock is successfully acquired, the business logic is executed, and the lock is deleted after the execution of the business is completed. Failed to create temporary node, blocking and waiting. Listen for the delete event. Once the temporary node is deleted, it indicates that the mutex operation is completed and you can try to acquire the lock again. Recursion: The process of acquiring a lock is a recursive operation, acquiring a lock-> listening-> acquiring a lock.
• How to avoid deadlock : The temporary node is created. When the service down session is closed, the temporary node will be deleted and the lock is automatically released.

Code

• Referring to the implementation of JDK lock plus the encapsulation of template method pattern, the encapsulation interface is as follows:

/**
 * @Description ZK分布式锁的接口
 * @Author 陈某
 * @Date 2020/4/7 22:52
 */
public interface ZKLock {
    /**
     * 获取锁
     */
    void lock() throws Exception;

    /**
     * 解锁
     */
    void unlock() throws Exception;
}

• The template abstract class is as follows:

/**
 * @Description 排他锁，模板类
 * @Author 陈某
 * @Date 2020/4/7 22:55
 */
public abstract class AbstractZKLockMutex implements ZKLock {

    /**
     * 节点路径
     */
    protected String lockPath;

    /**
     * zk客户端
     */
    protected CuratorFramework zkClient;

    private AbstractZKLockMutex(){}

    public AbstractZKLockMutex(String lockPath,CuratorFramework client){
        this.lockPath=lockPath;
        this.zkClient=client;
    }

    /**
     * 模板方法，搭建的获取锁的框架，具体逻辑交于子类实现
     * @throws Exception
     */
    @Override
    public final void lock() throws Exception {
        //获取锁成功
        if (tryLock()){
            System.out.println(Thread.currentThread().getName()+"获取锁成功");
        }else{  //获取锁失败
            //阻塞一直等待
            waitLock();
            //递归，再次获取锁
            lock();
        }
    }

    /**
     * 尝试获取锁，子类实现
     */
    protected abstract boolean tryLock() ;

    /**
     * 等待获取锁，子类实现
     */
    protected abstract void waitLock() throws Exception;

    /**
     * 解锁：删除节点或者直接断开连接
     */
    @Override
    public  abstract void unlock() throws Exception;
}

• The specific implementation class of the exclusive lock is as follows:

/**
 * @Description 排他锁的实现类，继承模板类 AbstractZKLockMutex
 * @Author 陈某
 * @Date 2020/4/7 23:23
 */
@Data
public class ZKLockMutex extends AbstractZKLockMutex {
​
    /**
     * 用于实现线程阻塞
     */
    private CountDownLatch countDownLatch;
​
    public ZKLockMutex(String lockPath,CuratorFramework zkClient){
        super(lockPath,zkClient);
    }
​
    /**
     * 尝试获取锁：直接创建一个临时节点，如果这个节点存在创建失败抛出异常，表示已经互斥了，
     * 反之创建成功
     * @throws Exception
     */
    @Override
    protected boolean tryLock()  {
        try {
            zkClient.create()
                    //临时节点
                    .withMode(CreateMode.EPHEMERAL)
                    //权限列表 world:anyone:crdwa
                    .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE)
                    .forPath(lockPath,"lock".getBytes());
            return true;
        }catch (Exception ex){
            return false;
        }
    }
​
​
    /**
     * 等待锁，一直阻塞监听
     * @return  成功获取锁返回true，反之返回false
     */
    @Override
    protected void waitLock() throws Exception {
        //监听节点的新增、更新、删除
        final NodeCache nodeCache = new NodeCache(zkClient, lockPath);
        //启动监听
        nodeCache.start();
        ListenerContainer<NodeCacheListener> listenable = nodeCache.getListenable();
​
        //监听器
        NodeCacheListener listener=()-> {
            //节点被删除，此时获取锁
            if (nodeCache.getCurrentData() == null) {
                //countDownLatch不为null，表示节点存在，此时监听到节点删除了，因此-1
                if (countDownLatch != null)
                    countDownLatch.countDown();
            }
        };
        //添加监听器
        listenable.addListener(listener);
​
        //判断节点是否存在
        Stat stat = zkClient.checkExists().forPath(lockPath);
        //节点存在
        if (stat!=null){
            countDownLatch=new CountDownLatch(1);
            //阻塞主线程，监听
            countDownLatch.await();
        }
        //移除监听器
        listenable.removeListener(listener);
    }
​
    /**
     * 解锁，直接删除节点
     * @throws Exception
     */
    @Override
    public void unlock() throws Exception {
        zkClient.delete().forPath(lockPath);
    }
}

How to design reentrant exclusive lock

• The reentrant logic is very simple, save a ConcurrentMap locally, key is the current thread, value is the defined data, the structure is as follows:

 private final ConcurrentMap<Thread, LockData> threadData = Maps.newConcurrentMap();

• The pseudo code for reentry is as follows:

public boolean tryLock(){
    //判断当前线程是否在threadData保存过
    //存在，直接return true
    //不存在执行获取锁的逻辑
    //获取成功保存在threadData中
}

Realization of read-write lock

• Read-write locks are divided into read locks and write locks, the difference is as follows: read locks allow multiple threads to read data at the same time, but do not allow the write thread to modify while reading. After the write lock is acquired, multiple threads are not allowed to write or read at the same time.
• How to implement read-write lock? There is a type of node in ZK called temporary ordered node, which was introduced above. Let's use the temporary ordered node to realize the function of read-write lock.

Design of read lock

• The read lock allows multiple threads to read at the same time, and does not allow threads to write while reading. The implementation principle is as follows:

 

• According to the above figure, acquiring a read lock is divided into the following steps: create a temporary ordered node (read lock owned by the current thread or called read node). Get all the child nodes in the path, and sort from small to large to get the neighboring write node (write lock) before the current node. If there is no nearby write node, the read lock is successfully acquired. If there is an adjacent write node, it monitors the delete event. Once the delete event is monitored, repeat steps 2 , 3 , 4 , and 5 (recursively) .

Write lock design

• Once a thread has acquired a write lock, no other threads are allowed to read or write. The principle of implementation is as follows:

 

• It can be seen from the above figure that the only thing that is different from the write lock is the listening node. Here is listening to the neighboring node (read node or write node). The read lock only needs to listen to the write node. The steps are as follows: create a temporary ordered node (the current thread owns ) 'S write lock or write node). Get all the child nodes under the path and sort them from small to large. Get the neighboring nodes (read nodes and write nodes) of the current node. If there are no neighboring nodes, the lock is successfully acquired. If there are neighboring nodes, monitor and delete them. Once the delete event is monitored, repeat steps 2 , 3 , 4 , and 5 (recursively) .

How to monitor

• Whether it is a write lock or a read lock, you need to listen to the previous node. The difference is that the read lock only listens to the adjacent write node. The write lock is to monitor all the neighboring nodes. The abstraction is actually a chain of listeners, as shown below:

 

• Each node is listening to the neighboring node in front of it. Once the previous node is deleted, it listens to the previous node after reordering, so that it recurs.

Code

• The author simply wrote the implementation of the read-write lock, which is first made and then optimized, and is not recommended for use in a production environment. code show as below:

public class ZKLockRW  {
​
    /**
     * 节点路径
     */
    protected String lockPath;
​
    /**
     * zk客户端
     */
    protected CuratorFramework zkClient;
​
    /**
     * 用于阻塞线程
     */
    private CountDownLatch countDownLatch=new CountDownLatch(1);
​
​
    private final static String WRITE_NAME="_W_LOCK";
​
    private final static String READ_NAME="_R_LOCK";
​
​
    public ZKLockRW(String lockPath, CuratorFramework client) {
        this.lockPath=lockPath;
        this.zkClient=client;
    }
​
    /**
     * 获取锁，如果获取失败一直阻塞
     * @throws Exception
     */
    public void lock() throws Exception {
        //创建节点
        String node = createNode();
        //阻塞等待获取锁
        tryLock(node);
        countDownLatch.await();
    }
​
    /**
     * 创建临时有序节点
     * @return
     * @throws Exception
     */
    private String createNode() throws Exception {
        //创建临时有序节点
       return zkClient.create()
                .withMode(CreateMode.EPHEMERAL_SEQUENTIAL)
                .withACL(ZooDefs.Ids.OPEN_ACL_UNSAFE)
                .forPath(lockPath);
    }
​
    /**
     * 获取写锁
     * @return
     */
    public  ZKLockRW writeLock(){
        return new ZKLockRW(lockPath+WRITE_NAME,zkClient);
    }
​
    /**
     * 获取读锁
     * @return
     */
    public  ZKLockRW readLock(){
        return new ZKLockRW(lockPath+READ_NAME,zkClient);
    }
​
    private void tryLock(String nodePath) throws Exception {
        //获取所有的子节点
        List<String> childPaths = zkClient.getChildren()
                .forPath("/")
                .stream().sorted().map(o->"/"+o).collect(Collectors.toList());
​
​
        //第一个节点就是当前的锁，直接获取锁。递归结束的条件
        if (nodePath.equals(childPaths.get(0))){
            countDownLatch.countDown();
            return;
        }
​
        //1. 读锁：监听最前面的写锁，写锁释放了，自然能够读了
        if (nodePath.contains(READ_NAME)){
            //查找临近的写锁
            String preNode = getNearWriteNode(childPaths, childPaths.indexOf(nodePath));
            if (preNode==null){
                countDownLatch.countDown();
                return;
            }
            NodeCache nodeCache=new NodeCache(zkClient,preNode);
            nodeCache.start();
            ListenerContainer<NodeCacheListener> listenable = nodeCache.getListenable();
            listenable.addListener(() -> {
                //节点删除事件
                if (nodeCache.getCurrentData()==null){
                    //继续监听前一个节点
                    String nearWriteNode = getNearWriteNode(childPaths, childPaths.indexOf(preNode));
                    if (nearWriteNode==null){
                        countDownLatch.countDown();
                        return;
                    }
                    tryLock(nearWriteNode);
                }
            });
        }
​
        //如果是写锁，前面无论是什么锁都不能读，直接循环监听上一个节点即可，直到前面无锁
        if (nodePath.contains(WRITE_NAME)){
            String preNode = childPaths.get(childPaths.indexOf(nodePath) - 1);
            NodeCache nodeCache=new NodeCache(zkClient,preNode);
            nodeCache.start();
            ListenerContainer<NodeCacheListener> listenable = nodeCache.getListenable();
            listenable.addListener(() -> {
                //节点删除事件
                if (nodeCache.getCurrentData()==null){
                    //继续监听前一个节点
                    tryLock(childPaths.get(childPaths.indexOf(preNode) - 1<0?0:childPaths.indexOf(preNode) - 1));
                }
            });
        }
    }
​
    /**
     * 查找临近的写节点
     * @param childPath 全部的子节点
     * @param index 右边界
     * @return
     */
    private String  getNearWriteNode(List<String> childPath,Integer index){
        for (int i = 0; i < index; i++) {
            String node = childPath.get(i);
            if (node.contains(WRITE_NAME))
                return node;
​
        }
        return null;
    }
​
}

Curator implements step-by-step lock

• Curator is a Zookeeper client open sourced by Netflix. Compared with the native client provided by Zookeeper, Curator has a higher level of abstraction, which simplifies the development of Zookeeper client.
• Curator has encapsulated it for us in terms of distributed locks, and the general implementation idea is implemented according to the author's above-mentioned idea. Small and medium-sized Internet companies still recommend using the framework package directly. After all, it is stable. Some large-scale Internet companies are handwritten.
• Creating an exclusive lock is simple, as follows:

//arg1：CuratorFramework连接对象，arg2：节点路径
lock=new InterProcessMutex(client,path);
//获取锁
lock.acquire();
//释放锁
lock.release();

• Please refer to the official documentation for more APIs, not the focus of this article.
• At this point, the introduction of ZK to implement distributed locks is finished. I have prepared some learning materials and videos for friends who are not familiar with Zookeeper. I have prepared some learning materials and videos for friends in need. get! ! !

A little benefit

For the friends who are not very familiar with Zookeeper, I have prepared some learning materials and videos for friends in need. The catalog is as follows:

Recommended reading: byte beating on three sides to get offers: network + IO + redis + JVM + GC + red-black tree + data structure