In many cases, we can see ZooKeeper in various framework applications, such as Kafka middleware, Dubbo framework, Hadoop and so on. Why do I see ZooKeeper everywhere?

basic knowledge

1. What is ZooKeeper

ZooKeeper is a distributed service coordination framework that provides a solution for distributed data consistency. Based on ZooKeeper's data structure, Watcher, and election mechanism, it can implement data publishing/subscribing, soft load balancing, naming services, and unified configuration. management, distributed locks, cluster management, and more.

The core implementation of Zookeeper is a distributed data storage system, which internally uses the ZAB protocol (Zookeeper Atomic Broadcast) for master-slave replication to ensure data consistency and reliability. In Zookeeper, data storage uses a data model called "Znode", which is similar to the Unix file system.

Znode is the most basic data unit in Zookeeper. It is a hierarchical tree structure. Each node has a path, where the root node is "/", and the child node path will add a relative path based on the parent node path , and finally form a complete tree structure.

In addition to Znode, Zookeeper also supports node monitoring and event mechanism. Monitoring can allow the client to respond to Znode changes in a timely manner, enhancing the real-time performance of the application.

2. Why use ZooKeeper

ZooKeeper can guarantee:

Update requests are made sequentially. Update requests from the same client are executed sequentially in the order they were sent
Data update atomicity. A data update either succeeds or fails
Globally unique data view . No matter which server the client connects to, the data view is consistent
real-time . Within a certain time range, the data read by the client is the latest

3. Data structure

The data structure of ZooKeeper is very similar to the Unix file system. It can be regarded as a tree in general. Each node is called a ZNode, and each ZNode can store 1M data by default . Each ZNode can be identified by a unique path . As shown below:

When creating a ZNode, the following four types can be specified, including:

PERSISTENT, persistent ZNode . After creation, even if the client disconnects from the server, it will not be deleted, and it will disappear only if the client actively deletes it.
PERSISTENT_SEQUENTIAL, persistent sequential number ZNode . Like persistent nodes, it will not be deleted after disconnection, and the number of ZNode will automatically increase.
EPHEMERAL, ephemeral ZNode . When the client disconnects from the server, the ZNode will be deleted.
EPEMERAL_SEQUENTIAL, temporary sequential number ZNode . Like ephemeral nodes, disconnects are deleted and the ZNode number is automatically incremented.

4. Monitoring notification mechanism

Watcher is a mechanism implemented based on the observer pattern . If we need to receive a notification when a ZNode node changes, we can use the Watcher listener.

The client registers with ZooKeeper the znode that needs to receive notifications by setting a watcher, and ZooKeeper will send a message to the client when the znode changes .

This notification mechanism is one-time . Once a watcher is triggered, ZooKeeper is removed from the corresponding storage. If you need to continuously monitor the changes of ZNode, you can set a new watcher to register with ZooKeeper after receiving the notification.

There are many types of monitoring points, such as monitoring ZNode data changes, monitoring ZNode child node changes, and monitoring ZNode creation or deletion .

5. Election mechanism

ZooKeeper is a highly available application framework because ZooKeeper supports clusters. When ZooKeeper is in the cluster state, the configuration file does not specify the Master and Slave, but internally elects when the ZooKeeper server is initialized, generating one as the Leader and multiple as Followers, and obeys the principle of half availability.

Due to the half-availability principle, the maximum allowable downtime for 5 servers and 6 servers is actually 3, so in order to save costs, the number of servers in the cluster is generally set to an odd number .

If at runtime, if the connection with the Leader cannot be maintained for a long time, an election will be conducted again to generate a new Leader to ensure the availability of the service .

use

1 download zookeeper

https://dlcdn.apache.org/zookeeper/zookeeper-3.8.1/apache-zookeeper-3.8.1-bin.tar.gz

2 modification

Copy the configuration file below and rename the newly copied one to zoo.cfg

Then change the file content to

tickTime=2000
dataDir=X:\\mygreensoftware\\apache-zookeeper-3.8.1-bin\\tmp\\data
dataLogDir=X:\\mygreensoftware\\apache-zookeeper-3.8.1-bin\\tmp\\logs
clientPort=2181

Create a directory for settings

Double click to start. .

3 Troubleshooting

wc, flash card! !

Don't worry, we may not configure JAVA_HOME well, let's enter the environment variable to configure JAVA_HOME

Note that this is the main path of jdk, without bin! !

ok, try to start again, well, there is no problem this time

Use in SpringBoot

Install the visualization plugin

the side

new connection

127.0.0.1:2181

then click connect

Dependency configuration

<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.6</version>
</dependency>

zookeeper: 
  server: 127.0.0.1:2181
  timeout: 3000

configuration class

package com.scm.springbootzookper.config;

import org.apache.zookeeper.ZooKeeper;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.io.IOException;

@Configuration
public class ZookeeperConfig {
    @Value("${zookeeper.server}")
    private String server;

    @Value("${zookeeper.timeout}")
    private Integer timeout;

    @Bean
    public ZooKeeper zkClient() throws IOException {
        return new ZooKeeper(server, timeout, watchedEvent -> {});
    }
}

Install httpclient for easy testing

Can refer to the following

https://blog.csdn.net/qq_53679247/article/details/130841001

CRUD

new controller

package com.scm.springbootzookper.controller;

import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooKeeper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

@RestController
@RequestMapping("/api")
public class ZookController {

    @Autowired
    ZooKeeper zkClient;

    @GetMapping("/zookeeper")
    public String getData() throws KeeperException, InterruptedException {
        String path = "/zookeeper";
        boolean watch = true;
        byte[] data = zkClient.getData(path, watch, null);
        return new String(data);
    }

}

Test with http client

create node

API

public String create(final String path, byte data[], List<ACL> acl, CreateMode createMode)

path ZNode path
data ZNode stored data
acl ACL permission control
createMode ZNode type

    @GetMapping("/addNode/{nodename}/{data}")
    public String addNode(@PathVariable("nodename")String nodename, @PathVariable("data") String data1){
        // 创建节点的路径
        String path = "/"+nodename;
        // 节点数据
        String data =data1;
        // 权限控制
        List<ACL> aclList = ZooDefs.Ids.OPEN_ACL_UNSAFE;
        // 创建节点的类型
        CreateMode createMode = CreateMode.PERSISTENT;

        String result = null;
        try {
            result = zkClient.create(path, data.getBytes(), aclList, createMode);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return result;
    }

httpclent environment data

{
  "dev": {
    "name": "value",
    "test": "test",
    "nodename": "node1",
    "data": "我是测试数据1"
  }
}

query node

    @GetMapping("/getData/{nodename}")
    public String getData(@PathVariable("nodename") String nodename){
        //数据的描述信息，包括版本号，ACL权限，子节点信息等等
        Stat stat = new Stat();
        //返回结果是byte[]数据，getData()方法底层会把描述信息复制到stat对象中
        byte[] bytes;
        String path="/"+nodename;
        try {
            bytes = zkClient.getData(path, false, stat);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        //打印结果
        System.out.println("ZNode的数据data:" + new String(bytes));//Hello World
        System.out.println("获取到dataVersion版本号:" + stat.getVersion());//默认数据版本号是0
        return new String(bytes);
    }

update node

For delete and update operations, the version number must be obtained before modification

    @GetMapping("/setData/{nodename}/{data}")
    public String setData(@PathVariable("nodename")String nodename, @PathVariable("data") String data1){
        String path = "/"+nodename;
        String data = data1;
        int version = 0;

        Stat stat = null;
        try {
            stat = zkClient.setData(path, data.getBytes(), version);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return stat.toString();
    }

delete node

    
    @GetMapping("/deleteNode/{nodename}")
    public String deleteNode(@PathVariable("nodename")String nodename){
        String path = "/"+nodename;
        int version = 0;
        try {
             zkClient.delete(path, version);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return "OK!";
    }

use monitor

new listener

package com.scm.springbootzookper.watch;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.springframework.stereotype.Component;

@Component
public class MyWatcher implements Watcher {
    @Override
    public void process(WatchedEvent watchedEvent) {
        Event.KeeperState state = watchedEvent.getState();
        Event.EventType type = watchedEvent.getType();
        System.out.println("检测到节点发生变化.....");
        System.out.println("节点名称:"+state.name());
        System.out.println("事件类型:"+type.name());
        System.out.println("节点路径"+watchedEvent.getPath());

    }
}

Change the method in the controller.

    @GetMapping("/setData/{nodename}/{data}")
    public String setData(@PathVariable("nodename")String nodename, @PathVariable("data") String data1) throws InterruptedException, KeeperException {
        String path = "/"+nodename;
        zkClient.exists(path, new MyWatcher());
        String data = data1;
        // 这里必须先拿到版本号才能更新
        int version =5;

        Stat stat = null;
        try {
            stat = zkClient.setData(path, data.getBytes(), version);
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
        return stat.toString();
    }

Use httpclient request, the result is as follows

Precautions

It should be noted that a registered listener can only be used once, and it will be invalid after use.

Executed serially. The process of client Watcher callback is a serial synchronization process, which is to ensure the order.

business use

Determine the change type of the node at the time of notification, and perform other operations.

package com.scm.springbootzookper.watch;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.springframework.stereotype.Component;

@Component
public class MyWatcher implements Watcher {
    @Override
    public void process(WatchedEvent watchedEvent) {
        Event.KeeperState state = watchedEvent.getState();
        Event.EventType type = watchedEvent.getType();
        if (Event.EventType.NodeDataChanged.getIntValue()==type.getIntValue()) {
            System.out.println("节点被修改了！");
        }
        if (Event.EventType.NodeDeleted.getIntValue()==type.getIntValue()) {
            System.out.println("节点被删除了！");
        }
        System.out.println("检测到节点发生变化.....");
        System.out.println("节点名称:"+state.name());
        System.out.println("事件类型:"+type.name());
        System.out.println("节点路径"+watchedEvent.getPath());

    }
}

Some business operations are possible.

The following is the source code of the Watch interface, we can notice the enumeration type,

//
// Source code recreated from a .class file by IntelliJ IDEA
// (powered by FernFlower decompiler)
//

package org.apache.zookeeper;

public interface Watcher {
    void process(WatchedEvent var1);

    public interface Event {
        public static enum EventType {
            None(-1),
            NodeCreated(1),
            NodeDeleted(2),
            NodeDataChanged(3),
            NodeChildrenChanged(4);

            private final int intValue;

            private EventType(int intValue) {
                this.intValue = intValue;
            }

            public int getIntValue() {
                return this.intValue;
            }

            public static EventType fromInt(int intValue) {
                switch (intValue) {
                    case -1:
                        return None;
                    case 0:
                    default:
                        throw new RuntimeException("Invalid integer value for conversion to EventType");
                    case 1:
                        return NodeCreated;
                    case 2:
                        return NodeDeleted;
                    case 3:
                        return NodeDataChanged;
                    case 4:
                        return NodeChildrenChanged;
                }
            }
        }

        public static enum KeeperState {
            /** @deprecated */
            @Deprecated
            Unknown(-1),
            Disconnected(0),
            /** @deprecated */
            @Deprecated
            NoSyncConnected(1),
            SyncConnected(3),
            AuthFailed(4),
            ConnectedReadOnly(5),
            SaslAuthenticated(6),
            Expired(-112);

            private final int intValue;

            private KeeperState(int intValue) {
                this.intValue = intValue;
            }

            public int getIntValue() {
                return this.intValue;
            }

            public static KeeperState fromInt(int intValue) {
                switch (intValue) {
                    case -112:
                        return Expired;
                    case -1:
                        return Unknown;
                    case 0:
                        return Disconnected;
                    case 1:
                        return NoSyncConnected;
                    case 3:
                        return SyncConnected;
                    case 4:
                        return AuthFailed;
                    case 5:
                        return ConnectedReadOnly;
                    case 6:
                        return SaslAuthenticated;
                    default:
                        throw new RuntimeException("Invalid integer value for conversion to KeeperState");
                }
            }
        }
    }
}

END.........

ZooKeeper quick start learning + application in springboot + business use of monitoring mechanism

foreword