Nacos--Source code analysis Sentinel Dashboard--Data persistence

1.0. Nacos source code analysis

There are many things worth learning about Nacos source code. In order to understand Nacos deeply , we analyze the source code and analyze the following two knowledge points:

1 : Nacos’s access principle to the registration center

2 : Nacos registration service process

Next, we will do an in-depth analysis of the Nacos source code. First, we will build the Nacos source code environment. The source code environment is relatively easy to build and almost no errors will be reported. We will not demonstrate the source code environment construction here.

The interaction between the client and the registration center server mainly focuses on service registration , service offline , service discovery , and subscription services . In fact, the most commonly used are service registration and service discovery . Below I will analyze these four functions from the perspective of source code.

In the Nacos source code, the com.alibaba.nacos.example. NamingExample class in nacos - example demonstrates the operation of these four functions respectively. We can use it as the entrance. The code is as follows:

public class NamingExample {
 public static void main(String[] args) throws NacosException {
 Properties properties = new Properties();
 properties.setProperty("serverAddr", System.getProperty("serverAddr"));
 properties.setProperty("namespace", System.getProperty("namespace"));
 //采用NamingService实现服务注册
 NamingService naming = NamingFactory.createNamingService(properties);
 //服务注册
 naming.registerInstance("nacos.test.3", "11.11.11.11", 8888, "TEST1");
 naming.registerInstance("nacos.test.3", "2.2.2.2", 9999, "DEFAULT");
 //服务发现System.out.println(naming.getAllInstances("nacos.test.3"));//服务下线naming.deregisterInstance("nacos.test.3", "2.2.2.2", 9999, "DEFAULT"); System.out.println(naming.getAllInstances("nacos.test.3" newExecutor executor =//Service subscription));





 ThreadPoolExecutor(1, 1, 0L,
 TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(),
 new ThreadFactory() {
 @Override
 public Thread newThread(Runnable r) {
 Thread thread = new Thread(r);
 thread.setName("test-thread");
 return thread;
 }
 }
 );
 naming.subscribe("nacos.test.3", new AbstractEventListener() {
 //EventListener onEvent is sync to handle, If process too low in onEvent, maybe block other onEvent callback.
 //So you can override getExecutor() to async handle event.
 @Override
 public Executor getExecutor() {
 return executor;
 }

 @Override
 public void onEvent(Event event) {
 System.out.println(((NamingEvent) event).getServiceName());
 System.out.println(((NamingEvent) event).getInstances());
 }
 });
 }
}

1.1.Nacos configuration storage mysql database

配置文件存储mysql，打开nacos-console工程下application.properties文件，添加mysql连接信息：

# 当前配置存储采用mysql存储
spring.datasource.platform=mysql

### Count of DB:
db.num=1

### Connect URL of DB:
db.url.0=jdbc:mysql://127.0.0.1:3306/nacos?characterEncoding=utf8&connectTimeout=1000

&socketTimeout=3000&autoReconnect=true&useUnicode=true&useSSL=false&serverTimezone=UTC
db.user.0=nacos
db.password.0=nacos

存储配置的表模型在nacos-console工程下resources/META-INF下schema.sql，在mysql库中执行脚本。

配置好上面的信息，nacos将会把配置文件信息保存到数据库中。

启动nacos，在nacos-console工程下，启动Nacos启动类。

@SpringBootApplication(scanBasePackages = "com.alibaba.nacos")
@ServletComponentScan
// 定时器
@EnableScheduling
public class Nacos {

    public static void main(String[] args) {
        SpringApplication.run(Nacos.class, args);
    }
}

1.2.客户端工作流程

我们先来看一下客户端是如何实现服务注册、服务发现、服务下线操作、服务订阅操作的。

1.2.1服务创建

一旦创建NamingService就会支持服务订阅。

NamingService对象创建功能：

/**
* 创建NamingService 对象
*/
public static NamingService createNamingService(Properties properties) throws NacosException {
 try {
 Class<?> driverImplClass = Class.forName("com.alibaba.nacos.client.naming.NacosNamingService");
 Constructor constructor = driverImplClass.getConstructor(Properties.class);
 // NamingService
 NamingService vendorImpl = (NamingService) constructor.newInstance(properties);
 return vendorImpl;
 } catch (Throwable e) {
 throw new NacosException(NacosException.CLIENT_INVALID_PARAM, e);
 }
}

接下来跟踪看看创建的NamingService功能，创建对象时会初始化远程代理对象、心跳检测和订阅定时任务。

/**
* 构造函数创建NacosNamingService对象
*/
public NacosNamingService(String serverList) throws NacosException {
    Properties properties = new Properties();
    properties.setProperty(PropertyKeyConst.SERVER_ADDR, serverList);
    // 初始化配置
    init(properties);
}

public NacosNamingService(Properties properties) throws NacosException {
    init(properties);
}

/**
* 初始胡过程
*/
private void init(Properties properties) throws NacosException {
    ValidatorUtils.checkInitParam(properties);
    this.namespace = InitUtils.initNamespaceForNaming(properties);
    InitUtils.initSerialization();
    initServerAddr(properties);
    InitUtils.initWebRootContext(properties);
    initCacheDir();
    initLogName(properties);
    // 创建代理对象执行远程操作 http协议
    this.serverProxy = new NamingProxy(this.namespace, this.endpoint, this.serverList, properties);
    // 定时任务心跳检测
    this.beatReactor = new BeatReactor(this.serverProxy, initClientBeatThreadCount(properties));
    // 订阅定时任务定时从服务拉取服务列表
    this.hostReactor = new HostReactor(this.serverProxy, beatReactor, this.cacheDir, isLoadCacheAtStart(properties),
            isPushEmptyProtect(properties), initPollingThreadCount(properties));
}

继续跟踪HostReactor订阅

public HostReactor(NamingProxy serverProxy, BeatReactor beatReactor, String cacheDir, boolean loadCacheAtStart,
 boolean pushEmptyProtection, int pollingThreadCount) {
 // init executorService
 this.executor = new ScheduledThreadPoolExecutor(pollingThreadCount, new ThreadFactory() {
 @Override
 public Thread newThread(Runnable r) {
 Thread thread = new Thread(r);
 thread.setDaemon(true);
 thread.setName("com.alibaba.nacos.client.naming.updater");
 return thread;
 }
 });

 this.beatReactor = beatReactor;
 this.serverProxy = serverProxy;
 this.cacheDir = cacheDir;
 if (loadCacheAtStart) {
 this.serviceInfoMap = new ConcurrentHashMap<String, ServiceInfo>(DiskCache.read(this.cacheDir));
 } else {
 this.serviceInfoMap = new ConcurrentHashMap<String, ServiceInfo>(16);
 }
 this.pushEmptyProtection = pushEmptyProtection;
 this.updatingMap = new ConcurrentHashMap<String, Object>();
 this.failoverReactor = new FailoverReactor(this, cacheDir);
 // go on 创建订阅服务列表对象
 this.pushReceiver = new PushReceiver(this);
 this.notifier = new InstancesChangeNotifier();

 NotifyCenter.registerToPublisher(InstancesChangeEvent.class, 16384);
 NotifyCenter.registerSubscriber(notifier);
}

跟踪订阅服务列表对象PushReceiver

public class PushReceiver implements Runnable, Closeable {

    private static final Charset UTF_8 = Charset.forName("UTF-8");

    private static final int UDP_MSS = 64 * 1024;
    // 定时任务线程池
    private ScheduledExecutorService executorService;

    private DatagramSocket udpSocket;

    private HostReactor hostReactor;

    private volatile boolean closed = false;

    public PushReceiver(HostReactor hostReactor) {
        try {
            this.hostReactor = hostReactor;
            // 通讯对象(NIO)->远程调用
            this.udpSocket = new DatagramSocket();
            // 创建定时任务线程池->定时任务
            this.executorService = new ScheduledThreadPoolExecutor(1, new ThreadFactory() {
                @Override
                public Thread newThread(Runnable r) {
                    Thread thread = new Thread(r);
                    thread.setDaemon(true);
                    thread.setName("com.alibaba.nacos.naming.push.receiver");
                    return thread;
                }
            });
            // 循环执行定时任务每过一段时间获取一次数据
            this.executorService.execute(this);
        } catch (Exception e) {
            NAMING_LOGGER.error("[NA] init udp socket failed", e);
        }
    }

    @Override
    public void run() {
        while (!closed) {
            try {

                // byte[] is initialized with 0 full filled by default
                byte[] buffer = new byte[UDP_MSS];
                DatagramPacket packet = new DatagramPacket(buffer, buffer.length);
                // 接收数据获取远程数据包 NIO
                udpSocket.receive(packet);

                String json = new String(IoUtils.tryDecompress(packet.getData()), UTF_8).trim();
                NAMING_LOGGER.info("received push data: " + json + " from " + packet.getAddress().toString());
                // 反序列化
                PushPacket pushPacket = JacksonUtils.toObj(json, PushPacket.class);
                String ack;
                if ("dom".equals(pushPacket.type) || "service".equals(pushPacket.type)) {
                    // 处理反序列化数据包
                    hostReactor.processServiceJson(pushPacket.data);
                    // send ack to server

// ack确认机制告诉服务器已经获取到信息了
                    ack = "{ \"type\": \"push-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime + "\", \"data\":"
                            + "\"\"}";
                } else if ("dump".equals(pushPacket.type)) {
                    // dump data to server
                    ack = "{ \"type\": \"dump-ack\"" + ", \"lastRefTime\": \"" + pushPacket.lastRefTime + "\", \"data\":"
                            + "\"" + StringUtils.escapeJavaScript(JacksonUtils.toJson(hostReactor.getServiceInfoMap()))
                            + "\"}";
                } else {
                    // do nothing send ack only
                    ack = "{ \"type\": \"unknown-ack\"" + ", \"lastRefTime\":\"" + pushPacket.lastRefTime
                            + "\", \"data\":" + "\"\"}";
                }

                udpSocket.send(new DatagramPacket(ack.getBytes(UTF_8), ack.getBytes(UTF_8).length,
                        packet.getSocketAddress()));
            } catch (Exception e) {
                if (closed) {
                    return;
                }
                NAMING_LOGGER.error("[NA] error while receiving push data", e);
            }
        }
    }

... ...

继续处理远程拉取的服务信息，反序列化数据包，将反序列化数据包存储到serviceInfoMap中。

public ServiceInfo processServiceJson(String json) {
 // 解析数据包
 ServiceInfo serviceInfo = JacksonUtils.toObj(json, ServiceInfo.class);
 ServiceInfo oldService = serviceInfoMap.get(serviceInfo.getKey());

 if (pushEmptyProtection && !serviceInfo.validate()) {
 //empty or error push, just ignore
 return oldService;
 }

 boolean changed = false;

 if (oldService != null) {

 if (oldService.getLastRefTime() > serviceInfo.getLastRefTime()) {
 NAMING_LOGGER.warn("out of date data received, old-t: " + oldService.getLastRefTime() + ", new-t: "
 + serviceInfo.getLastRefTime());
 }
 // ===获取服务数据存储本地serviceInfoMap中===
 serviceInfoMap.put(serviceInfo.getKey(), serviceInfo);

 Map<String, Instance> oldHostMap = new HashMap<String, Instance>(oldService.getHosts().size());
 for (Instance host : oldService.getHosts()) {
 oldHostMap.put(host.toInetAddr(), host);
 }

... ...

总结，创建NamigService对象时做了几件事：

1. 创建代理对象，执行远程操作。

2. 创建定时任务执行心跳检测，定时向服务发送心跳检测。

3. 创建订阅定时任务，定时从服务拉取服务列表信息。

A. 创建通讯对象(NIO协议)->执行远程调用。

B. 创建定时任务线程池->执行定时任务，每过一段时间从服务拉取一次服务信息。将拉取的服务信息反序列化后存入本地的缓存(Map<String, ServiceInfo> serviceInfoMap)。

C. 服务信息变更时，会拉取服务信息存储到serviceInfoMap中。

1.2.2.服务注册

我们沿着案例中的服务注册方法调用找到nacos-api中的NamingService.registerInstance() 并找到它的实现类和方法 com.alibaba.nacos.client.naming.NacosNamingService，代码如下：

/***
* 服务注册
* @param serviceName 服务名字
* @param ip 服务IP
* @param port 服务端口
* @param clusterName 集群名字
* @throws NacosException
*/
@Override
public void registerInstance(String serviceName, String ip, int port, String clusterName) throws NacosException {
    registerInstance(serviceName, Constants.DEFAULT_GROUP, ip, port, clusterName);
}

@Override
public void registerInstance(String serviceName, String groupName, String ip, int port, String clusterName) throws NacosException {
    //设置实例IP：Port，默认权重为1.0
    Instance instance = new Instance();
    instance.setIp(ip);
    instance.setPort(port);
    instance.setWeight(1.0);
    instance.setClusterName(clusterName);
    //注册实例
    registerInstance(serviceName, groupName, instance);
}

@Override
public void registerInstance(String serviceName, Instance instance) throws NacosException {
    registerInstance(serviceName, Constants.DEFAULT_GROUP, instance);
}

/***
* 实例注册
* @param serviceName name of service
* @param groupName group of service
* @param instance instance to register
* @throws NacosException
*/
@Override
public void registerInstance(String serviceName, String groupName, Instance instance) throws NacosException {
    NamingUtils.checkInstanceIsLegal(instance);
    String groupedServiceName = NamingUtils.getGroupedName(serviceName, groupName);
    // 该字段表示注册的实例是否是临时实例还是持久化实例。
    // 如果是临时实例，则不会在 Nacos 服务端持久化存储，需要通过上报心跳的方式进行包活，
    // 如果一段时间内没有上报心跳，则会被 Nacos 服务端摘除。
    if (instance.isEphemeral()) {
        // 为注册服务设置一个定时任务获取心跳信息，默认为5s汇报一次
        // 心跳检测任务   服务名+@+组名作为key执行心跳检测
        BeatInfo beatInfo = beatReactor.buildBeatInfo(groupedServiceName, instance);
        beatReactor.addBeatInfo(groupedServiceName, beatInfo);
    }
    // 注册到服务端 serverProxy在NamingServer创建的时候创建代理对象执行远程调用
    serverProxy.registerService(groupedServiceName, groupName, instance);
}

注册主要做了两件事:

第一件事：为注册的服务设置一个定时任务，定时拉取服务信息。

第二件事：将服务注册到服务端。

1:启动一个定时任务，定时拉取服务信息，时间间隔为5s,如果拉下来服务正常，不做处理，如果不正常，重新注册

2:发送http请求给注册中心服务端，调用服务注册接口，注册服务

上面代码我们可以看到定时任务添加，但并未完全看到远程请求，serverProxy.registerService()方法如下，会先封装请求参数，接下来调用reqApi() 而reqApi()最后会调用callServer() ，代码如下：

public void registerService(String serviceName, String groupName, Instance instance) throws NacosException {
 NAMING_LOGGER.info("[REGISTER-SERVICE] {} registering service {} with instance: {}", namespaceId, serviceName, instance);
 //封装Http请求参数
 final Map<String, String> params = new HashMap<String, String>(16);
 params.put(CommonParams.NAMESPACE_ID, namespaceId);
 params.put(CommonParams.SERVICE_NAME, serviceName);
 params.put(CommonParams.GROUP_NAME, groupName);
 params.put(CommonParams.CLUSTER_NAME, instance.getClusterName());
 params.put("ip", instance.getIp());
 params.put("port", String.valueOf(instance.getPort()));
 params.put("weight", String.valueOf(instance.getWeight()));
 params.put("enable", String.valueOf(instance.isEnabled()));
 params.put("healthy", String.valueOf(instance.isHealthy()));
 params.put("ephemeral", String.valueOf(instance.isEphemeral()));
 params.put("metadata", JacksonUtils.toJson(instance.getMetadata()));
 //执行Http请求
 reqApi(UtilAndComs.nacosUrlInstance, params, HttpMethod.POST);
}

/**
* Request api.
* 请求API
*
* @param api api
* @param params parameters
* @param body body
* @param servers servers
* @param method http method
* @return result
* @throws NacosException nacos exception
*/
public String reqApi(String api, Map<String, String> params, Map<String, String> body, List<String> servers,
 String method) throws NacosException {

 params.put(CommonParams.NAMESPACE_ID, getNamespaceId());

 if (CollectionUtils.isEmpty(servers) && StringUtils.isBlank(nacosDomain)) {
 throw new NacosException(NacosException.INVALID_PARAM, "no server available");
 }

 NacosException exception = new NacosException();

 if (StringUtils.isNotBlank(nacosDomain)) {
 for (int i = 0; i < maxRetry; i++) {
 try {
 // 远程调用入口
 return callServer(api, params, body, nacosDomain, method);
 } catch (NacosException e) {
 exception = e;
 if (NAMING_LOGGER.isDebugEnabled()) {
 NAMING_LOGGER.debug("request {} failed.", nacosDomain, e);
 }
 }
 }
 } else {
 Random random = new Random(System.currentTimeMillis());
 int index = random.nextInt(servers.size());

 for (int i = 0; i < servers.size(); i++) {
 String server = servers.get(index);
 try {
 return callServer(api, params, body, server, method);
 } catch (NacosException e) {
 exception = e;
 if (NAMING_LOGGER.isDebugEnabled()) {
 NAMING_LOGGER.debug("request {} failed.", server, e);
 }
 }
 index = (index + 1) % servers.size();
 }
 }

 NAMING_LOGGER.error("request: {} failed, servers: {}, code: {}, msg: {}", api, servers, exception.getErrCode(),
 exception.getErrMsg());

 throw new NacosException(exception.getErrCode(),
 "failed to req API:" + api + " after all servers(" + servers + ") tried: " + exception.getMessage());

}

/***
*执行远程调用
**/
public String callServer(String api, Map<String, String> params, Map<String, String> body, String curServer,
 String method) throws NacosException {
 long start = System.currentTimeMillis();
 long end = 0;
 injectSecurityInfo(params);
 //封装请求头部
 Header header = builderHeader();
 //请求是Http还是Https协议
 String url;
 if (curServer.startsWith(UtilAndComs.HTTPS) || curServer.startsWith(UtilAndComs.HTTP)) {
 url = curServer + api;
 } else {
 if (!IPUtil.containsPort(curServer)) {
 curServer = curServer + IPUtil.IP_PORT_SPLITER + serverPort;
 }
 url = NamingHttpClientManager.getInstance().getPrefix() + curServer + api;
 }
 try {
 //执行远程请求,并获取结果集
 HttpRestResult<String> restResult = nacosRestTemplate.exchangeForm(url, header, Query.newInstance().initParams(params), body, method, String.class);
 end = System.currentTimeMillis();
 MetricsMonitor.getNamingRequestMonitor(method, url, String.valueOf(restResult.getCode())).observe(end - start);
 //结果集解析
 if (restResult.ok()) {
 return restResult.getData();
 }
 if (HttpStatus.SC_NOT_MODIFIED == restResult.getCode()) {
 return StringUtils.EMPTY;
 }
 throw new NacosException(restResult.getCode(), restResult.getMessage());
 } catch (Exception e) {
 NAMING_LOGGER.error("[NA] failed to request", e);
 throw new NacosException(NacosException.SERVER_ERROR, e);
 }
}

执行远程Http请求的对象是NacosRestTemplate，该对象就是封装了普通的Http请求。

/**
* Execute the HTTP method to the given URI template, writing the given request entity to the request, and returns
* the response as { @link HttpRestResult}.
*
* @param url url
* @param header http header param
* @param query http query param 查询条件封装
* @param bodyValues http body param
* @param httpMethod http method
* @param responseType return type
* @return { @link HttpRestResult}
* @throws Exception ex
*/
public <T> HttpRestResult<T> exchangeForm(String url, Header header, Query query, Map<String, String> bodyValues,
 String httpMethod, Type responseType) throws Exception {
 RequestHttpEntity requestHttpEntity = new RequestHttpEntity(
 header.setContentType(MediaType.APPLICATION_FORM_URLENCODED), query, bodyValues);
 return execute(url, httpMethod, requestHttpEntity, responseType);
}

执行远程调用请求使用http协议

/**
* 执行远程请求
*
* @param url
* @param httpMethod
* @param requestEntity
* @param responseType
* @param <T>
* @return
* @throws Exception
*/
@SuppressWarnings("unchecked")
private <T> HttpRestResult<T> execute(String url, String httpMethod, RequestHttpEntity requestEntity,
 Type responseType) throws Exception {
 // url：http://127.0.0.1:8848/nacos/v1/ns/instance
 URI uri = HttpUtils.buildUri(url, requestEntity.getQuery());
 if (logger.isDebugEnabled()) {
 logger.debug("HTTP method: {}, url: {}, body: {}", httpMethod, uri, requestEntity.getBody());
 }

 ResponseHandler<T> responseHandler = super.selectResponseHandler(responseType);
 HttpClientResponse response = null;
 try {
 // HttpClientRequest执行远程调用
 response = this.requestClient().execute(uri, httpMethod, requestEntity);
 return responseHandler.handle(response);
 } finally {
 if (response != null) {
 response.close();
 }
 }
}

/**
* 获取HttpClientRequest
*/
private HttpClientRequest requestClient() {
 if (CollectionUtils.isNotEmpty(interceptors)) {
 if (logger.isDebugEnabled()) {
 logger.debug("Execute via interceptors :{}", interceptors);
 }
 return new InterceptingHttpClientRequest(requestClient, interceptors.iterator());
 }
 return requestClient;
}

这是远程调用使用的是JdkHttpClientRequest发起的远程调用

@Override
public HttpClientResponse execute(URI uri, String httpMethod, RequestHttpEntity requestHttpEntity)
 throws Exception {
 final Object body = requestHttpEntity.getBody();
 final Header headers = requestHttpEntity.getHeaders();
 replaceDefaultConfig(requestHttpEntity.getHttpClientConfig());

 HttpURLConnection conn = (HttpURLConnection) uri.toURL().openConnection();
 Map<String, String> headerMap = headers.getHeader();
 if (headerMap != null && headerMap.size() > 0) {
 for (Map.Entry<String, String> entry : headerMap.entrySet()) {
 conn.setRequestProperty(entry.getKey(), entry.getValue());
 }
 }

 conn.setConnectTimeout(this.httpClientConfig.getConTimeOutMillis());
 conn.setReadTimeout(this.httpClientConfig.getReadTimeOutMillis());
 conn.setRequestMethod(httpMethod);
 if (body != null && !"".equals(body)) {
 String contentType = headers.getValue(HttpHeaderConsts.CONTENT_TYPE);
 String bodyStr = JacksonUtils.toJson(body);
 if (MediaType.APPLICATION_FORM_URLENCODED.equals(contentType)) {
 Map<String, String> map = JacksonUtils.toObj(bodyStr, HashMap.class);
 bodyStr = HttpUtils.encodingParams(map, headers.getCharset());
 }
 if (bodyStr != null) {
 conn.setDoOutput(true);
 byte[] b = bodyStr.getBytes();
 conn.setRequestProperty("Content-Length", String.valueOf(b.length));
 // 获取远程网络输入流执行远程请求
 OutputStream outputStream = conn.getOutputStream();
 outputStream.write(b, 0, b.length);
 outputStream.flush();
 IoUtils.closeQuietly(outputStream);
 }
 }
 conn.connect();
 return new JdkHttpClientResponse(conn);
}

远程调用，调用的是nacos的服务信息。远程掉用url：http://127.0.0.1:8848/nacos/v1/ns/instance
执行Http请求/nacos为服务根地址，在nacos-console配置文件中配置。

#*************** Spring Boot Related Configurations ***************#
### Default web context path:默认web服务根地址
server.servlet.contextPath=/nacos
### Default web server port:
server.port=8848

远程调用post请求/v1/ns/instance为nacos服务地址，在nacos-naming包中。

HealthController 执行心跳检测服务

/**
* Health status related operation controller.
* 心跳相关控制器
* @author nkorange
* @author nanamikon
* @since 0.8.0
*/
@RestController("namingHealthController")
@RequestMapping(UtilsAndCommons.NACOS_NAMING_CONTEXT + "/health")
public class HealthController {

... ...

}

InstanceController 执行服务实例注册服务

/**
* Instance operation controller.
* 服务实例控制
* @author nkorange
*/
@RestController
// /vi/ns/instance
@RequestMapping(UtilsAndCommons.NACOS_NAMING_CONTEXT + "/instance")
public class InstanceController {

/**
* Register new instance.
*
* @param request http request
* @return 'ok' if success
* @throws Exception any error during register
*/
@CanDistro //Distro协议(数据临时一致性协议)
@PostMapping
@Secured(parser = NamingResourceParser.class, action = ActionTypes.WRITE)
public String register(HttpServletRequest request) throws Exception {

    final String namespaceId = WebUtils
            .optional(request, CommonParams.NAMESPACE_ID, Constants.DEFAULT_NAMESPACE_ID);
    final String serviceName = WebUtils.required(request, CommonParams.SERVICE_NAME);
    NamingUtils.checkServiceNameFormat(serviceName);

    final Instance instance = parseInstance(request);
    // 注册服务实例
    serviceManager.registerInstance(namespaceId, serviceName, instance);
    return "ok";
}

}

1.2.3.服务发现

我们沿着案例中的服务发现方法调用找到nacos-api中的NamingService.getAllInstances() 并找到它的实现类和方法com.alibaba.nacos.client.naming.NacosNamingService.getAllInstances() ，代码如下：

@Override
public List<Instance> getAllInstances(String serviceName, String groupName, List<String> clusters,
 boolean subscribe) throws NacosException {

 ServiceInfo serviceInfo;
 if (subscribe) {
 // 开启服务订阅则从本地获取服务列表本地服务列表存储在serviceInfoMap中
 // 服务订阅会将服务列表定时更新存储到serviceInfoMao中
 serviceInfo = hostReactor.getServiceInfo(NamingUtils.getGroupedName(serviceName, groupName),
 StringUtils.join(clusters, ","));
 } else {
 // 没有开启服务订阅从远端获取请求nacos服务发起远程调用获取服务信息
 serviceInfo = hostReactor
 .getServiceInfoDirectlyFromServer(NamingUtils.getGroupedName(serviceName, groupName),
 StringUtils.join(clusters, ","));
 }
 List<Instance> list;
 if (serviceInfo == null || CollectionUtils.isEmpty(list = serviceInfo.getHosts())) {
 return new ArrayList<Instance>();
 }
 return list;
}

上面的代码调用了hostReactor.getServiceInfo() 方法。

public ServiceInfo getServiceInfo(final String serviceName, final String clusters) {
    NAMING_LOGGER.debug("failover-mode: " + failoverReactor.isFailoverSwitch());
    String key = ServiceInfo.getKey(serviceName, clusters);
    if (failoverReactor.isFailoverSwitch()) {
        return failoverReactor.getService(key);
    }
    /*1。先从本地缓存中获取服务对象，因为启动是第一次进来，所以缓存站不存在*/
    ServiceInfo serviceObj = getServiceInfo0(serviceName, clusters);
    if (null == serviceObj) {
        /*构建服务实例*/
        serviceObj = new ServiceInfo(serviceName, clusters);
        /*将服务实例存放到缓存中*/
        serviceInfoMap.put(serviceObj.getKey(), serviceObj);
        /*更新nacos-上的服务*/
        updatingMap.put(serviceName, new Object());
        /*主动获取，并且更新到缓存本地，以及已过期的服务更新等*/
        updateServiceNow(serviceName, clusters);
        updatingMap.remove(serviceName);
    } else if (updatingMap.containsKey(serviceName)) {
        if (UPDATE_HOLD_INTERVAL > 0) {
            updateServiceNow(serviceName, clusters);
            // hold a moment waiting for update finish
            synchronized (serviceObj) {
                try {
                    serviceObj.wait(UPDATE_HOLD_INTERVAL);
                } catch (InterruptedException e) {
                    NAMING_LOGGER
                            .error("[getServiceInfo] serviceName:" + serviceName + ", clusters:" + clusters, e);
                }
            }
        }
    }
    /*2.开启定时任务*/
    scheduleUpdateIfAbsent(serviceName, clusters);
    return serviceInfoMap.get(serviceObj.getKey());
}

该方法会先调用 getServiceInfo0() 方法从本地缓存获取数据。

private ServiceInfo getServiceInfo0(String serviceName, String clusters) {
    // 获取对应的key值 Group@服务名格式
    String key = ServiceInfo.getKey(serviceName, clusters);
    // 获取key对应的缓存这里的值是NamingService执行定时任务获取的
    return serviceInfoMap.get(key);
}

缓存没有数据，就构建实例更新到Nacos，并从Nacos中获取最新数据。updateServiceNow(serviceName, clusters);主动从远程服务器获取更新数据。

private void updateServiceNow(String serviceName, String clusters) {
    try {
        updateService(serviceName, clusters);
    } catch (NacosException e) {
        NAMING_LOGGER.error("[NA] failed to update serviceName: " + serviceName, e);
    }
}

最终会调用updateService()方法，在该方法中完成远程请求和数据处理，源码如下：

/**
* Update service now.
*
* @param serviceName service name
* @param clusters    clusters
*/
public void updateService(String serviceName, String clusters) throws NacosException {
    // 从本地缓存列表获取服务
    ServiceInfo oldService = getServiceInfo0(serviceName, clusters);
    try {
        // 代理发起http请求远程调用获取服务以及提供者端口信息，端口等
        String result = serverProxy.queryList(serviceName, clusters, pushReceiver.getUdpPort(), false);

        if (StringUtils.isNotEmpty(result)) {
            // 反序列化服务信息并存储到serviceInfoMap中
            processServiceJson(result);
        }
    } finally {
        if (oldService != null) {
            synchronized (oldService) {
                oldService.notifyAll();
            }
        }
    }
}

getServiceInfo0()方法在前面已经介绍过了。processServiceJson()方法在服务创建介绍过。

回到开头，没有开启服务订阅会从远端获取，请求nacos服务发起远程调用获取服务信息。调用getServiceInfoDirectlyFromServer()方法。

/**
* 直接从nacos服务获取服务信息
*/
public ServiceInfo getServiceInfoDirectlyFromServer(final String serviceName, final String clusters)
        throws NacosException {
    String result = serverProxy.queryList(serviceName, clusters, 0, false);
    if (StringUtils.isNotEmpty(result)) {
        return JacksonUtils.toObj(result, ServiceInfo.class);
    }
    return null;
}

这里仍然是NamingService对象创建的代理对象发起远程调用获取服务信息queryList()。

/**
* Query instance list.
*/
public String queryList(String serviceName, String clusters, int udpPort, boolean healthyOnly)
 throws NacosException {

 final Map<String, String> params = new HashMap<String, String>(8);
 params.put(CommonParams.NAMESPACE_ID, namespaceId);
 params.put(CommonParams.SERVICE_NAME, serviceName);
 params.put("clusters", clusters);
 params.put("udpPort", String.valueOf(udpPort));
 params.put("clientIP", NetUtils.localIP());
 params.put("healthyOnly", String.valueOf(healthyOnly));

 return reqApi(UtilAndComs.nacosUrlBase + "/instance/list", params, HttpMethod.GET);
}

reqApi()在服务注册环节已经介绍过了。

1.2.4.服务下线

我们沿着案例中的服务下线方法调用找到nacos-api中的NamingService.deregisterInstance()并找到它的实现类和方法 NacosNamingService.deregisterInstance()，代码如下：

@Override
public void deregisterInstance(String serviceName, String groupName, String ip, int port, String clusterName) throws NacosException {
    //构建实例信息
    Instance instance = new Instance();
    instance.setIp(ip);
    instance.setPort(port);
    instance.setClusterName(clusterName);
    //服务下线操作
    deregisterInstance(serviceName, groupName, instance);
}

@Override
public void deregisterInstance(String serviceName, String groupName, Instance instance) throws NacosException {
    if (instance.isEphemeral()) {
        //移除心跳信息监测的定时任务
        beatReactor.removeBeatInfo(NamingUtils.getGroupedName(serviceName, groupName), instance.getIp(),
                instance.getPort());
    }
    //发送远程请求执行服务下线销毁操作
    serverProxy.deregisterService(NamingUtils.getGroupedName(serviceName, groupName), instance);
}

发起远程delete请求。

/**
* deregister instance from a service.
*
* @param serviceName name of service
* @param instance instance
* @throws NacosException nacos exception
*/
public void deregisterService(String serviceName, Instance instance) throws NacosException {

 NAMING_LOGGER
 .info("[DEREGISTER-SERVICE] {} deregistering service {} with instance: {}", namespaceId, serviceName,
 instance);

 final Map<String, String> params = new HashMap<String, String>(8);
 params.put(CommonParams.NAMESPACE_ID, namespaceId);
 params.put(CommonParams.SERVICE_NAME, serviceName);
 params.put(CommonParams.CLUSTER_NAME, instance.getClusterName());
 params.put("ip", instance.getIp());
 params.put("port", String.valueOf(instance.getPort()));
 params.put("ephemeral", String.valueOf(instance.isEphemeral()));

 reqApi(UtilAndComs.nacosUrlInstance, params, HttpMethod.DELETE);
}

这里会远程调用nacos-naming服务InstanceController接口中的delete方法销毁服务。

/**
* Deregister instances.
*
* @param request http request
* @return 'ok' if success
* @throws Exception any error during deregister
*/
@CanDistro
@DeleteMapping
@Secured(parser = NamingResourceParser.class, action = ActionTypes.WRITE)
public String deregister(HttpServletRequest request) throws Exception {
    Instance instance = getIpAddress(request);
    String namespaceId = WebUtils.optional(request, CommonParams.NAMESPACE_ID, Constants.DEFAULT_NAMESPACE_ID);
    String serviceName = WebUtils.required(request, CommonParams.SERVICE_NAME);
    NamingUtils.checkServiceNameFormat(serviceName);

    Service service = serviceManager.getService(namespaceId, serviceName);
    if (service == null) {
        Loggers.SRV_LOG.warn("remove instance from non-exist service: {}", serviceName);
        return "ok";
    }
    // 移除服务数据
    serviceManager.removeInstance(namespaceId, serviceName, instance.isEphemeral(), instance);
    return "ok";
}

服务下线方法比较简单，和服务注册做的事情正好相反，也做了两件事，第一件事：不在进行心跳检测。第二件事：请求服务端服务下线接口。

1.2.4.服务订阅

我们可以查看订阅服务的案例，会先创建一个线程池，接下来会把线程池封装到监听器中，而监听器中可以监听指定实例信息，代码如下：

//服务订阅
Executor executor = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new
 LinkedBlockingQueue<Runnable>(), new ThreadFactory() {
 @Override
 public Thread newThread(Runnable r) {
 Thread thread = new Thread(r);
 thread.setName("test-thread");
 return thread;
 }
});
naming.subscribe("nacos.test.3", new AbstractEventListener() {
 //EventListener onEvent is sync to handle, If process too low in onEvent,maybe block other onEvent callback.
 //So you can override getExecutor() to async handle event.
 @Override
 public Executor getExecutor() {
 return executor;
 }

 //读取监听到的服务实例
 @Override
 public void onEvent(Event event) {
 System.out.println(((NamingEvent) event).getServiceName());
 System.out.println(((NamingEvent) event).getInstances());
 }
});

我们沿着案例中的服务订阅方法调用找到nacos-api中的NamingService.subscribe()并找到它的实现类和方法NacosNamingService.deregisterInstance()，代码如下：

public void subscribe(String serviceName, String clusters, EventListener eventListener) {
    //注册监听
    notifier.registerListener(serviceName, clusters, eventListener);
    //获取并更新服务实例
    getServiceInfo(serviceName, clusters);
}

此时会注册监听，注册监听就是将当前的监听对象信息注入到listenerMap集合中，在监听对象的指定方法onEvent中可以读取实例信息，代码如下：

public void registerListener(String serviceName, String clusters, EventListener listener) {
 String key = ServiceInfo.getKey(serviceName, clusters);
 ConcurrentHashSet<EventListener> eventListeners = listenerMap.get(key);
 if (eventListeners == null) {
 synchronized (lock) {
 eventListeners = listenerMap.get(key);
 if (eventListeners == null) {
 eventListeners = new ConcurrentHashSet<EventListener>();
 listenerMap.put(key, eventListeners);
 }
 }
 }
 //将当前监听对象放入到集合中，在监听对象的onEvent中可以读出对应的实例对象
 eventListeners.add(listener);
}

getServiceInfo(serviceName, clusters)获取服务实例，先从本地缓存获取，本地获取不到就从服务器获取，前面服务发现已经介绍过了。

1.3.服务端工作流程

注册中心服务端的主要功能包括，接收客户端的服务注册，服务发现，服务下线的功能，但是除了这些和客户端的交互之外，服务端还要做一些更重要的事情，就是我们常常会在分布式系统中听到的AP和CP，作为一个集群，nacos即实现了AP也实现了CP，其中AP使用的自己实现的Distro协议，而CP是采用raft协议实现的，这个过程中牵涉到心跳、选主等操作。

我们来学习一下注册中心服务端接收客户端服务注册的功能。

1.3.1.注册处理

我们先来学习一下Nacos的工具类WebUtils ，该工具类在nacos-core工程下，该工具类是用于处理请求参数转化的，里面提供了2个常被用到的方法required()和optional()：

required方法通过参数名key，解析HttpServletRequest请求中的参数，并转码为UTF-8编码。

optional方法在required方法的基础上增加了默认值，如果获取不到，则返回默认值。

代码如下：

/**
* required方法通过参数名key，解析HttpServletRequest请求中的参数，并转码为UTF-8编码。
*/
public static String required(final HttpServletRequest req, final String key) {
    String value = req.getParameter(key);
    if (StringUtils.isEmpty(value)) {
        throw new IllegalArgumentException("Param '" + key + "' is required.");
    }
    String encoding = req.getParameter("encoding");
    return resolveValue(value, encoding);
}

/**
* optional方法在required方法的基础上增加了默认值，如果获取不到，则返回默认值。
*/
public static String optional(final HttpServletRequest req, final String key, final String defaultValue) {
    if (!req.getParameterMap().containsKey(key) ||
            req.getParameterMap().get(key)[0] == null) {
        return defaultValue;
    }
    String value = req.getParameter(key);
    if (StringUtils.isBlank(value)) {
        return defaultValue;
    }
    String encoding = req.getParameter("encoding");
    return resolveValue(value, encoding);
}

nacos server-client使用了http协议来交互，那么在server端必定提供了http接口的入口，并且在core模块看到其依赖了spring boot starter，所以它的http接口由集成了Spring的web服务器支持，简单地说就是像我们平时写的业务服务一样，有controller层和service层。

以OpenAPI作为入口来学习，我们找到 /nacos/v1/ns/instance服务注册接口，在nacos-naming工程中我们可以看到InstanceController正是我们要找的对象，如下图：

处理服务注册，我们直接找对应的POST方法即可，代码如下：

/**
* Register new instance.
* 接收客户端注册信息
*
* @param request http request
* @return 'ok' if success
* @throws Exception any error during register
*/
@CanDistro
@PostMapping
@Secured(parser = NamingResourceParser.class, action = ActionTypes.WRITE)
public String register(HttpServletRequest request) throws Exception {
    //获取namespaceid，该参数是可选参数
    final String namespaceId = WebUtils.optional(request, CommonParams.NAMESPACE_ID, Constants.DEFAULT_NAMESPACE_ID);
    //获取服务名字
    final String serviceName = WebUtils.required(request, CommonParams.SERVICE_NAME);
    //校验服务的名字，服务的名字格式为groupName@@serviceName
    NamingUtils.checkServiceNameFormat(serviceName);
    //创建实例
    final Instance instance = parseInstance(request);
    //注册服务
    serviceManager.registerInstance(namespaceId, serviceName, instance);
    return "ok";
}

如上图，该方法主要用于接收客户端注册信息，并且会校验参数是否存在问题，如果不存在问题就创建服务的实例，服务实例创建后将服务实例注册到Nacos中，注册的方法如下：

public void registerInstance(String namespaceId, String serviceName, Instance instance) throws NacosException {
    //判断本地缓存中是否存在该命名空间，如果不存在就创建，之后判断该命名空间下是否
    //存在该服务，如果不存在就创建空的服务
    //如果实例为空，则创建实例，并且会将创建的实例存入到serviceMap集合中
    createEmptyService(namespaceId, serviceName, instance.isEphemeral());
    //从serviceMap集合中获取创建的实例
    Service service = getService(namespaceId, serviceName);
    if (service == null) {
        throw new NacosException(NacosException.INVALID_PARAM, "service not found, namespace: " + namespaceId + ", service: " + serviceName);
    }
    //服务注册，这一步才会把服务的实例信息和服务绑定起来
    addInstance(namespaceId, serviceName, instance.isEphemeral(), instance);
}

注册的方法中会先创建该实例对象，创建前先检查本地缓存是否存在该实例对象，如果不存在就创建，最后注册该服务，并且该服务会和实例信息捆绑到一起，并将信息同步到磁盘，数据同步到磁盘就涉及到数据一致性了，我们接下来讲解Nacos的数据一致性。

1.3.2.一致性算法Distro协议介绍

Distro是阿里巴巴的私有协议，目前流行的Nacos服务管理框架就采用了Distro协议。Distro协议被定位为临时数据的一致性协议：该类型协议，不需要把数据存储到磁盘或者数据库，因为临时数据通常和服务器保持一个session会话，该会话只要存在，数据就不会丢失。

Distro协议保证写必须永远是成功的，即使可能会发生网络分区。当网络恢复时，把各数据分片的数据进行合并。

Distro协议具有以下特点：

1:专门为了注册中心而创造出的协议；

2:客户端与服务端有两个重要的交互，服务注册与心跳发送；

3:客户端以服务为维度向服务端注册，注册后每隔一段时间向服务端发送一次心跳，心跳包需要带上注册服

务的全部信息，在客户端看来，服务端节点对等，所以请求的节点是随机的；

4:客户端请求失败则换一个节点重新发送请求；

5:服务端节点都存储所有数据，但每个节点只负责其中一部分服务，在接收到客户端的“写”(注册、心

跳、下线等)请求后，服务端节点判断请求的服务是否为自己负责，如果是，则处理，否则交由负责的节点

处理；

6:每个服务端节点主动发送健康检查到其他节点，响应的节点被该节点视为健康节点；

7:服务端在接收到客户端的服务心跳后，如果该服务不存在，则将该心跳请求当做注册请求来处理；

8:服务端如果长时间未收到客户端心跳，则下线该服务；

9:负责的节点在接收到服务注册、服务心跳等写请求后将数据写入后即返回，后台异步地将数据同步给其他

节点；

10:节点在收到读请求后直接从本机获取后返回，无论数据是否为最新。

1.3.3.Distro服务启动-寻址模式

Distro协议服务端节点发现使用寻址机制来实现服务端节点的管理。在Nacos中，寻址模式有三种：

单机模式(StandaloneMemberLookup)

文件模式(FileConfigMemberLookup)

服务器模式(AddressServerMemberLookup)

三种寻址模式如下图：

在com.alibaba.nacos.core.cluster.lookup.LookupFactory中有创建寻址方式，可以创建集群启动方式、单机启动方式，不同启动方式就决定了不同寻址模式。

/**
* Create the target addressing pattern.
* 创建寻址模式
*
* @param memberManager { @link ServerMemberManager}
* @return { @link MemberLookup}
* @throws NacosException NacosException
*/
public static MemberLookup createLookUp(ServerMemberManager memberManager) throws NacosException {
    //NacosServer 集群方式启动
    if (!EnvUtil.getStandaloneMode()) {
        String lookupType = EnvUtil.getProperty(LOOKUP_MODE_TYPE);
        //由参数中传入的寻址方式得到LookupType对象
        LookupType type = chooseLookup(lookupType);
        //选择寻址方式
        LOOK_UP = find(type);
        //设置当前寻址方式
        currentLookupType = type;
    } else {
        //NacosServer单机启动
        LOOK_UP = new StandaloneMemberLookup();
    }
    LOOK_UP.injectMemberManager(memberManager);
    Loggers.CLUSTER.info("Current addressing mode selection : {}",
            LOOK_UP.getClass().getSimpleName());
    return LOOK_UP;
}

/***
* 选择寻址方式
* @param type
* @return
*/
private static MemberLookup find(LookupType type) {
    //文件寻址模式，也就是配置cluster.conf配置文件将多个节点串联起来，
    // 通过配置文件寻找其他节点，以达到和其他节点通信的目的
    if (LookupType.FILE_CONFIG.equals(type)) {
        LOOK_UP = new FileConfigMemberLookup();
        return LOOK_UP;
    }
    //服务器模式
    if (LookupType.ADDRESS_SERVER.equals(type)) {
        LOOK_UP = new AddressServerMemberLookup();
        return LOOK_UP;
    }
    // unpossible to run here
    throw new IllegalArgumentException();
}

单节点寻址模式会直接创建StandaloneMemberLookup对象，而文件寻址模式会创建FileConfigMemberLookup对象，服务器寻址模式会创建AddressServerMemberLookup；

1.3.3.1.单机寻址模式

单机模式直接寻找自己的IP:PORT地址。

public class StandaloneMemberLookup extends AbstractMemberLookup {

    @Override
    public void start() {
        if (start.compareAndSet(false, true)) {
            // 获取自己的IP:port
            String url = InetUtils.getSelfIP() + ":" + EnvUtil.getPort();
            afterLookup(MemberUtil.readServerConf(Collections.singletonList(url)));
        }
    }
}

1.3.3.2.文件寻址模式

文件寻址模式主要在创建集群的时候，通过cluster.conf 来配置集群，程序可以通过监听cluster.conf 文件变化实现动态管理节点，FileConfigMemberLookup源码如下：

public class FileConfigMemberLookup extends AbstractMemberLookup {
 //创建文件监听器
 private FileWatcher watcher = new FileWatcher() {
 //文件发生变更事件
 @Override
 public void onChange(FileChangeEvent event) {
 readClusterConfFromDisk();
 }

 //检查context是否包含cluster.conf
 @Override
 public boolean interest(String context) {
 return StringUtils.contains(context, "cluster.conf");
 }
 };

 @Override
 public void start() throws NacosException {
 if (start.compareAndSet(false, true)) {
 readClusterConfFromDisk();
 // 使用inotify机制来监视文件更改，并自动触发对cluster.conf的读取
 try {
 WatchFileCenter.registerWatcher(EnvUtil.getConfPath(), watcher);
 } catch (Throwable e) {
 Loggers.CLUSTER.error("An exception occurred in the launch file monitor : {}", e.getMessage());
 }
 }
 }

 @Override
 public void destroy() throws NacosException {
 WatchFileCenter.deregisterWatcher(EnvUtil.getConfPath(), watcher);
 }

 private void readClusterConfFromDisk() {
 Collection<Member> tmpMembers = new ArrayList<>();
 try {
 List<String> tmp = EnvUtil.readClusterConf();
 tmpMembers = MemberUtil.readServerConf(tmp);
 } catch (Throwable e) {
 Loggers.CLUSTER.error("nacos-XXXX [serverlist] failed to get serverlist from disk!, error : {}", e.getMessage());
 }
 afterLookup(tmpMembers);
 }
}

1.3.3.3.服务器寻址模式

使用地址服务器存储节点信息，会创建AddressServerMemberLookup，服务端定时拉取信息进行管理；

public class AddressServerMemberLookup extends AbstractMemberLookup {
 private final GenericType<RestResult<String>> genericType =
 new GenericType<RestResult<String>>() {
 };
 public String domainName;
 public String addressPort;
 public String addressUrl;
 public String envIdUrl;
 public String addressServerUrl;
 private volatile boolean isAddressServerHealth = true;
 private int addressServerFailCount = 0;
 private int maxFailCount = 12;
 private final NacosRestTemplate restTemplate =
 HttpClientBeanHolder.getNacosRestTemplate(Loggers.CORE);
 private volatile boolean shutdown = false;

 @Override
 public void start() throws NacosException {
 if (start.compareAndSet(false, true)) {
 this.maxFailCount = Integer.parseInt(EnvUtil.getProperty("maxHealthCheckFailCount", "12"));
 initAddressSys();
 run();
 }
 }

 /***
 * 获取服务器地址
 */
 private void initAddressSys() {
 String envDomainName = System.getenv("address_server_domain");
 if (StringUtils.isBlank(envDomainName)) {
 domainName = EnvUtil.getProperty("address.server.domain", "jmenv.tbsite.net");
 } else {
 domainName = envDomainName;
 }
 String envAddressPort = System.getenv("address_server_port");
 if (StringUtils.isBlank(envAddressPort)) {
 addressPort = EnvUtil.getProperty("address.server.port", "8080");
 } else {
 addressPort = envAddressPort;
 }
 String envAddressUrl = System.getenv("address_server_url");
 if (StringUtils.isBlank(envAddressUrl)) {
 addressUrl = EnvUtil.getProperty("address.server.url", EnvUtil.getContextPath() + "/" + "serverlist");
 } else {
 addressUrl = envAddressUrl;
 }
 addressServerUrl = "http://" + domainName + ":" + addressPort + addressUrl;
 envIdUrl = "http://" + domainName + ":" + addressPort + "/env";
 Loggers.CORE.info("ServerListService address-server port:" + addressPort);
 Loggers.CORE.info("ADDRESS_SERVER_URL:" + addressServerUrl);
 }

 @SuppressWarnings("PMD.UndefineMagicConstantRule")
 private void run() throws NacosException {
 // With the address server, you need to perform a synchronous member node pull at startup
 // Repeat three times, successfully jump out
 boolean success = false;
 Throwable ex = null;
 int maxRetry = EnvUtil.getProperty("nacos.core.address-server.retry", Integer.class, 5);
 for (int i = 0; i < maxRetry; i++) {
 try {
 //拉取集群节点信息
 syncFromAddressUrl();
 success = true;
 break;
 } catch (Throwable e) {
 ex = e;
 Loggers.CLUSTER.error("[serverlist] exception, error : {}", ExceptionUtil.getAllExceptionMsg(ex));
 }
 }
 if (!success) {
 throw new NacosException(NacosException.SERVER_ERROR, ex);
 }
 //创建定时任务
 GlobalExecutor.scheduleByCommon(new AddressServerSyncTask(), 5_000L);
 }

 @Override
 public void destroy() throws NacosException {
 shutdown = true;
 }

 @Override
 public Map<String, Object> info() {
 Map<String, Object> info = new HashMap<>(4);
 info.put("addressServerHealth", isAddressServerHealth);
 info.put("addressServerUrl", addressServerUrl);
 info.put("envIdUrl", envIdUrl);
 info.put("addressServerFailCount", addressServerFailCount);
 return info;
 }

 private void syncFromAddressUrl() throws Exception {
 RestResult<String> result = restTemplate.get(addressServerUrl, Header.EMPTY, Query.EMPTY, genericType.getType());
 if (result.ok()) {
 isAddressServerHealth = true;
 Reader reader = new StringReader(result.getData());
 try {
 afterLookup(MemberUtil.readServerConf(EnvUtil.analyzeClusterConf(reader)));
 } catch (Throwable e) {
 Loggers.CLUSTER.error("[serverlist] exception for analyzeClusterConf, error : {}", ExceptionUtil.getAllExceptionMsg(e));
 }
 addressServerFailCount = 0;
 } else {
 addressServerFailCount++;
 if (addressServerFailCount >= maxFailCount) {
 isAddressServerHealth = false;
 }
 Loggers.CLUSTER.error("[serverlist] failed to get serverlist, error code {}", result.getCode());
 }
 }

 // 定时任务
 class AddressServerSyncTask implements Runnable {
 @Override
 public void run() {
 if (shutdown) {
 return;
 }
 try {
 //拉取服务列表
 syncFromAddressUrl();
 } catch (Throwable ex) {
 addressServerFailCount++;
 if (addressServerFailCount >= maxFailCount) {
 isAddressServerHealth = false;
 }
 Loggers.CLUSTER.error("[serverlist] exception, error : {}", ExceptionUtil.getAllExceptionMsg(ex));
 } finally {
 GlobalExecutor.scheduleByCommon(this, 5_000L);
 }
 }
 }
}

1.3.5.集群数据同步

Nacos数据同步分为全量同步和增量同步，所谓全量同步就是初始化数据一次性同步，而增量同步是指有数据增加的时候，只同步增加的数据。

1.3.5.1.全量同步

全量同步流程比较复杂，流程如上图：

1:启动一个定时任务线程DistroLoadDataTask加载数据，调用load()方法加载数据

2:调用loadAllDataSnapshotFromRemote()方法从远程机器同步所有的数据

3:从namingProxy代理获取所有的数据data

4:构造http请求，调用httpGet方法从指定的server获取数据

5:从获取的结果result中获取数据bytes

6:处理数据processData

7:从data反序列化出datumMap

8:把数据存储到dataStore，也就是本地缓存dataMap

9:监听器不包括key，就创建一个空的service，并且绑定监听器

10:监听器listener执行成功后，就更新data store

◆ 任务启动

在com.alibaba.nacos.core.distributed.distro.DistroProtocol的构造函数中调用startDistroTask()方法，该方法会执行startVerifyTask()和startLoadTask() ，我们重点关注startLoadTask() ,该方法代码如下：

/***
* 启动DistroTask
*/
private void startDistroTask() {
    if (EnvUtil.getStandaloneMode()) {
        isInitialized = true;
        return;
    }
    //启动startVerifyTask，做数据同步校验
    startVerifyTask();
    //启动DistroLoadDataTask，批量加载数据
    startLoadTask();
}

//启动DistroLoadDataTask
private void startLoadTask() {
    //处理状态回调对象
    DistroCallback loadCallback = new DistroCallback() {
        //处理成功
        @Override
        public void onSuccess() {
            isInitialized = true;
        }

        //处理失败
        @Override
        public void onFailed(Throwable throwable) {
            isInitialized = false;
        }
    };
    //执行DistroLoadDataTask，是一个多线程
    GlobalExecutor.submitLoadDataTask(new DistroLoadDataTask(memberManager, distroComponentHolder, distroConfig, loadCallback));
}

/***
* 启动startVerifyTask
* 数据校验
*/
private void startVerifyTask() {
    GlobalExecutor.schedulePartitionDataTimedSync(
            new DistroVerifyTask(memberManager, distroComponentHolder), distroConfig.getVerifyIntervalMillis());
}

数据校验

public class DistroVerifyTask implements Runnable {

 private final ServerMemberManager serverMemberManager;

 private final DistroComponentHolder distroComponentHolder;

 public DistroVerifyTask(ServerMemberManager serverMemberManager, DistroComponentHolder distroComponentHolder) {
 this.serverMemberManager = serverMemberManager;
 this.distroComponentHolder = distroComponentHolder;
 }

 @Override
 public void run() {
 try {
 // 获取集群中所有节点
 List<Member> targetServer = serverMemberManager.allMembersWithoutSelf();
 if (Loggers.DISTRO.isDebugEnabled()) {
 Loggers.DISTRO.debug("server list is: {}", targetServer);
 }
 for (String each : distroComponentHolder.getDataStorageTypes()) {
 // 同步数据校验
 verifyForDataStorage(each, targetServer);
 }
 } catch (Exception e) {
 Loggers.DISTRO.error("[DISTRO-FAILED] verify task failed.", e);
 }
 }

 private void verifyForDataStorage(String type, List<Member> targetServer) {
 DistroData distroData = distroComponentHolder.findDataStorage(type).getVerifyData();
 if (null == distroData) {
 return;
 }
 distroData.setType(DataOperation.VERIFY);
 for (Member member : targetServer) {
 try {
 // 同步数据校验
 distroComponentHolder.findTransportAgent(type).syncVerifyData(distroData, member.getAddress());
 } catch (Exception e) {
 Loggers.DISTRO.error(String
 .format("[DISTRO-FAILED] verify data for type %s to %s failed.", type, member.getAddress()), e);
 }
 }
 }
}

执行校验

@Override
public boolean syncVerifyData(DistroData verifyData, String targetServer) {
    if (!memberManager.hasMember(targetServer)) {
        return true;
    }
    NamingProxy.syncCheckSums(verifyData.getContent(), targetServer);
    return true;
}

/**
* 同步检查总结
*/
public static void syncCheckSums(byte[] checksums, String server) {
 try {
 Map<String, String> headers = new HashMap<>(128);

 headers.put(HttpHeaderConsts.CLIENT_VERSION_HEADER, VersionUtils.version);
 headers.put(HttpHeaderConsts.USER_AGENT_HEADER, UtilsAndCommons.SERVER_VERSION);
 headers.put(HttpHeaderConsts.CONNECTION, "Keep-Alive");

 HttpClient.asyncHttpPutLarge(
 "http://" + server + EnvUtil.getContextPath() + UtilsAndCommons.NACOS_NAMING_CONTEXT
 + TIMESTAMP_SYNC_URL + "?source=" + NetUtils.localServer(), headers, checksums,
 new Callback<String>() {
 @Override
 public void onReceive(RestResult<String> result) {
 if (!result.ok()) {
 Loggers.DISTRO.error("failed to req API: {}, code: {}, msg: {}",
 "http://" + server + EnvUtil.getContextPath()
 + UtilsAndCommons.NACOS_NAMING_CONTEXT + TIMESTAMP_SYNC_URL,
 result.getCode(), result.getMessage());
 }
 }

 @Override
 public void onError(Throwable throwable) {
 Loggers.DISTRO.error("failed to req API:" + "http://" + server + EnvUtil.getContextPath()
 + UtilsAndCommons.NACOS_NAMING_CONTEXT + TIMESTAMP_SYNC_URL, throwable);
 }

 @Override
 public void onCancel() {

 }
 });
 } catch (Exception e) {
 Loggers.DISTRO.warn("NamingProxy", e);
 }
}

◆ 数据执行加载

上面方法会调用DistroLoadDataTask对象，而该对象其实是个线程，因此会执行它的run方法，run方法会调用load()方法实现数据全量加载，代码如下：

/***
* 数据加载过程
*/
@Override
public void run() {
    try {
        //加载数据
        load();
        if (!checkCompleted()) {
            GlobalExecutor.submitLoadDataTask(this, distroConfig.getLoadDataRetryDelayMillis());
        } else {
            loadCallback.onSuccess();
            Loggers.DISTRO.info("[DISTRO-INIT] load snapshot data success");
        }
    } catch (Exception e) {
        loadCallback.onFailed(e);
        Loggers.DISTRO.error("[DISTRO-INIT] load snapshot data failed. ", e);
    }
}

/***
* 加载数据，并同步
* @throws Exception
*/
private void load() throws Exception {
    while (memberManager.allMembersWithoutSelf().isEmpty()) {
        Loggers.DISTRO.info("[DISTRO-INIT] waiting server list init...");
        TimeUnit.SECONDS.sleep(1);
    }
    while (distroComponentHolder.getDataStorageTypes().isEmpty()) {
        Loggers.DISTRO.info("[DISTRO-INIT] waiting distro data storage register...");
        TimeUnit.SECONDS.sleep(1);
    }
    //同步数据
    for (String each : distroComponentHolder.getDataStorageTypes()) {
        if (!loadCompletedMap.containsKey(each) || !loadCompletedMap.get(each)) {
            //从远程机器上同步所有数据
            loadCompletedMap.put(each, loadAllDataSnapshotFromRemote(each));
        }
    }
}

◆ 数据同步

数据同步会通过Http请求从远程服务器获取数据，并同步到当前服务的缓存中，执行流程如下：

1:loadAllDataSnapshotFromRemote()从远程加载所有数据，并处理同步到本机

2:transportAgent.getDatumSnapshot()远程加载数据，通过Http请求执行远程加载

3:dataProcessor.processSnapshot()处理数据同步到本地

数据处理完整逻辑代码如下：loadAllDataSnapshotFromRemote()方法:

/***
* 从远程机器上同步所有数据
*/
private boolean loadAllDataSnapshotFromRemote(String resourceType) {
    DistroTransportAgent transportAgent = distroComponentHolder.findTransportAgent(resourceType);
    DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
    if (null == transportAgent || null == dataProcessor) {
        Loggers.DISTRO.warn("[DISTRO-INIT] Can't find component for type {}, transportAgent: {}, dataProcessor: {}",
                resourceType, transportAgent, dataProcessor);
        return false;
    }
    //遍历集群成员节点,不包括自己
    for (Member each : memberManager.allMembersWithoutSelf()) {
        try {
            Loggers.DISTRO.info("[DISTRO-INIT] load snapshot {} from {}", resourceType, each.getAddress());
            //从远程节点加载数据，调用http请求接口: distro/datums;
            DistroData distroData = transportAgent.getDatumSnapshot(each.getAddress());
            //处理数据
            boolean result = dataProcessor.processSnapshot(distroData);
            Loggers.DISTRO.info("[DISTRO-INIT] load snapshot {} from {} result: {}", resourceType, each.getAddress(), result);
            if (result) {
                return true;
            }
        } catch (Exception e) {
            Loggers.DISTRO.error("[DISTRO-INIT] load snapshot {} from {} failed.", resourceType, each.getAddress(), e);
        }
    }
    return false;
}

远程加载数据代码如下：transportAgent.getDatumSnapshot() 方法

/***
* 从namingProxy代理获取所有的数据data，从获取的结果result中获取数据bytes；
* @param targetServer target server.
* @return
*/
@Override
public DistroData getDatumSnapshot(String targetServer) {
 try {
 //从namingProxy代理获取所有的数据data，从获取的结果result中获取数据bytes；
 byte[] allDatum = NamingProxy.getAllData(targetServer);
 //将数据封装成DistroData
 return new DistroData(new DistroKey("snapshot", KeyBuilder.INSTANCE_LIST_KEY_PREFIX), allDatum);
 } catch (Exception e) {
 throw new DistroException(String.format("Get snapshot from %s failed.", targetServer), e);
 }
}

/**
* Get all datum from target server.
* NamingProxy.getAllData
* 执行HttpGet请求，并获取返回数据
*
* @param server target server address
* @return all datum byte array
* @throws Exception exception
*/
public static byte[] getAllData(String server) throws Exception {
 //参数封装
 Map<String, String> params = new HashMap<>(8);
 //组装URL，并执行HttpGet请求，获取结果集
 RestResult<String> result = HttpClient.httpGet("http://" + server + EnvUtil.getContextPath() +
 UtilsAndCommons.NACOS_NAMING_CONTEXT + ALL_DATA_GET_URL, new ArrayList<>(), params);
 //返回数据
 if (result.ok()) {
 return result.getData().getBytes();
 }
 throw new IOException("failed to req API: " + "http://"
 + server + EnvUtil.getContextPath()
 + UtilsAndCommons.NACOS_NAMING_CONTEXT
 + ALL_DATA_GET_URL + ". code:"
 + result.getCode() + " msg: "
 + result.getMessage());
}

处理数据同步到本地

@Override
public boolean processSnapshot(DistroData distroData) {
    try {
        return processData(distroData.getContent());
    } catch (Exception e) {
        return false;
    }
}

dataProcessor.processSnapshot()

/**
* 数据处理并更新本地缓存
*
* @param data
* @return
* @throws Exception
*/
private boolean processData(byte[] data) throws Exception {
 if (data.length > 0) {
 //从data反序列化出datumMap
 Map<String, Datum<Instances>> datumMap = serializer.deserializeMap(data, Instances.class);
 // 把数据存储到dataStore，也就是本地缓存dataMap
 for (Map.Entry<String, Datum<Instances>> entry : datumMap.entrySet()) {
 dataStore.put(entry.getKey(), entry.getValue());
 //监听器不包括key，就创建一个空的service，并且绑定监听器
 if (!listeners.containsKey(entry.getKey())) {
 // pretty sure the service not exist:
 if (switchDomain.isDefaultInstanceEphemeral()) {
 // create empty service
 //创建一个空的service
 Loggers.DISTRO.info("creating service {}", entry.getKey());
 Service service = new Service();
 String serviceName = KeyBuilder.getServiceName(entry.getKey());
 String namespaceId = KeyBuilder.getNamespace(entry.getKey());
 service.setName(serviceName);
 service.setNamespaceId(namespaceId);
 service.setGroupName(Constants.DEFAULT_GROUP);
 // now validate the service. if failed, exception will be thrown
 service.setLastModifiedMillis(System.currentTimeMillis());
 service.recalculateChecksum();
 // The Listener corresponding to the key value must not be empty
 // 与键值对应的监听器不能为空,这里的监听器类型是 ServiceManager
 RecordListener listener = listeners.get(KeyBuilder.SERVICE_META_KEY_PREFIX).peek();
 if (Objects.isNull(listener)) {
 return false;
 }
 //为空的绑定监听器
 listener.onChange(KeyBuilder.buildServiceMetaKey(namespaceId, serviceName), service);
 }
 }
 }
 //循环所有datumMap
 for (Map.Entry<String, Datum<Instances>> entry : datumMap.entrySet()) {
 if (!listeners.containsKey(entry.getKey())) {
 // Should not happen:
 Loggers.DISTRO.warn("listener of {} not found.", entry.getKey());
 continue;
 }
 try {
 //执行监听器的onChange监听方法
 for (RecordListener listener : listeners.get(entry.getKey())) {
 listener.onChange(entry.getKey(), entry.getValue().value);
 }
 } catch (Exception e) {
 Loggers.DISTRO.error("[NACOS-DISTRO] error while execute listener of key: {}", entry.getKey(), e);
 continue;
 }
 // Update data store if listener executed successfully:
 // 监听器listener执行成功后，就更新dataStore
 dataStore.put(entry.getKey(), entry.getValue());
 }
 }
 return true;
}

到此实现数据全量同步，其实全量同步最终封装的协议还是Http。

1.3.5.2.增量同步

新增数据使用异步广播同步：

1:DistroProtocol 使用 sync() 方法接收增量数据

2:向其他节点发布广播任务

调用 distroTaskEngineHolder 发布延迟任务

3:调用 DistroDelayTaskProcessor.process() 方法进行任务投递：将延迟任务转换为异步变更任务

4:执行变更任务 DistroSyncChangeTask.run() 方法：向指定节点发送消息

调用 DistroHttpAgent.syncData() 方法发送数据

调用 NamingProxy.syncData() 方法发送数据

5:异常任务调用 handleFailedTask() 方法进行处理

调用 DistroFailedTaskHandler 处理失败任务

调用 DistroHttpCombinedKeyTaskFailedHandler 将失败任务重新投递成延迟任务。

◆ 增量数据入口

我们回到服务注册，服务注册的InstanceController.register() 就是数据入口，它会调用ServiceManager.registerInstance()，执行数据同步的时候，调用addInstance() ,在该方法中会执行 DistroConsistencyServiceImpl.put()，该方法是增量同步的入口，会调用distroProtocol.sync()方法，代码如下：

/***
* 数据保存
* @param key key of data, this key should be globally unique
* @param value value of data
* @throws NacosException
*/
@Overridesync()
public void put(String key, Record value) throws NacosException {
    //将数据存入到dataStore中
    onPut(key, value);
    //使用distroProtocol同步数据
    distroProtocol.sync(new DistroKey(key, KeyBuilder.INSTANCE_LIST_KEY_PREFIX),
            DataOperation.CHANGE, globalConfig.getTaskDispatchPeriod() / 2);
}

sync()方法会执行任务发布，代码如下：

public void sync(DistroKey distroKey, DataOperation action, long delay) {
    //向除了自己外的所有节点广播
    for (Member each : memberManager.allMembersWithoutSelf()) {
        DistroKey distroKeyWithTarget = new DistroKey(distroKey.getResourceKey(), distroKey.getResourceType(), each.getAddress());
        DistroDelayTask distroDelayTask = new DistroDelayTask(distroKeyWithTarget, action, delay);
        //从distroTaskEngineHolder获取延时执行引擎，并将distroDelayTask任务添加进来
        //执行延时任务发布
        distroTaskEngineHolder.getDelayTaskExecuteEngine().
                addTask(distroKeyWithTarget, distroDelayTask);
        if (Loggers.DISTRO.isDebugEnabled()) {
            Loggers.DISTRO.debug("[DISTRO-SCHEDULE] {} to {}", distroKey, each.getAddress());
        }
    }
}

◆ 增量同步操作

延迟任务对象我们可以从DistroTaskEngineHolder构造函数中得知是DistroDelayTaskProcessor，代码如下：

/***
* 构造函数指定任务处理器
* @param distroComponentHolder
*/
public DistroTaskEngineHolder(DistroComponentHolder distroComponentHolder) {
    DistroDelayTaskProcessor defaultDelayTaskProcessor = new DistroDelayTaskProcessor(this, distroComponentHolder);
    //指定任务处理器defaultDelayTaskProcessor
    delayTaskExecuteEngine.setDefaultTaskProcessor(defaultDelayTaskProcessor);
}

它延迟执行的时候会执行process方法，该方法正是执行数据同步的地方，它会执行DistroSyncChangeTask任务，代码如下：

/***
* 任务处理过程
* @param task task.
* @return
*/
@Override
public boolean process(NacosTask task) {
    if (!(task instanceof DistroDelayTask)) {
        return true;
    }
    DistroDelayTask distroDelayTask = (DistroDelayTask) task;
    DistroKey distroKey = distroDelayTask.getDistroKey();
    if (DataOperation.CHANGE.equals(distroDelayTask.getAction())) {
        //将延迟任务变更成异步任务，异步任务对象是一个线程
        DistroSyncChangeTask syncChangeTask = new DistroSyncChangeTask(distroKey, distroComponentHolder);
        //将任务添加到NacosExecuteTaskExecuteEngine中，并执行
        distroTaskEngineHolder.getExecuteWorkersManager().addTask(distroKey, syncChangeTask);
        return true;
    }
    return false;
}

DistroSyncChangeTask实质上是任务的开始，它自身是一个线程，所以会执行它的run方法，而run方法这是数据同步操作，代码如下：

/***
* 执行数据同步
*/
@Override
public void run() {
    Loggers.DISTRO.info("[DISTRO-START] {}", toString());
    try {
        //获取本地缓存数据
        String type = getDistroKey().getResourceType();
        DistroData distroData = distroComponentHolder.findDataStorage(type).getDistroData(getDistroKey());
        distroData.setType(DataOperation.CHANGE);
        //向其他节点同步数据
        boolean result = distroComponentHolder.findTransportAgent(type).syncData(distroData, getDistroKey().getTargetServer());
        if (!result) {
            handleFailedTask();
        }
        Loggers.DISTRO.info("[DISTRO-END] {} result: {}", toString(), result);
    } catch (Exception e) {
        Loggers.DISTRO.warn("[DISTRO] Sync data change failed.", e);
        handleFailedTask();
    }
}

数据同步会执行调用syncData，该方法其实就是通过Http协议将数据发送到其他节点实现数据同步，代码如下：

/***
* 向其他节点同步数据
* @param data data
* @param targetServer target server
* @return
*/
@Override
public boolean syncData(DistroData data, String targetServer) {
    if (!memberManager.hasMember(targetServer)) {
        return true;
    }
    //获取数据字节数组
    byte[] dataContent = data.getContent();
    //通过Http协议同步数据
    return NamingProxy.syncData(dataContent, data.getDistroKey().getTargetServer());
}

◆ 详细增量数据同步(单机)

Distro一致性算法，临时数据一致性数据结构DataStore，临时数据会被存储到这个数据结构中。

/**
* Store of data.
* Distro 临时数据一致性数据存储结构
* @author nkorange
* @since 1.0.0
*/
@Component
public class DataStore {

 private Map<String, Datum> dataMap = new ConcurrentHashMap<>(1024);

 public void put(String key, Datum value) {
 dataMap.put(key, value);
 }

 public Datum remove(String key) {
 return dataMap.remove(key);
 }

 public Set<String> keys() {
 return dataMap.keySet();
 }

 public Datum get(String key) {
 return dataMap.get(key);
 }

 public boolean contains(String key) {
 return dataMap.containsKey(key);
 }

 /**
 * Batch get datum for a list of keys.
 *
 * @param keys of datum
 * @return list of datum
 */
 public Map<String, Datum> batchGet(List<String> keys) {
 Map<String, Datum> map = new HashMap<>(128);
 for (String key : keys) {
 Datum datum = dataMap.get(key);
 if (datum == null) {
 continue;
 }
 map.put(key, datum);
 }
 return map;
 }

 public int getInstanceCount() {
 int count = 0;
 for (Map.Entry<String, Datum> entry : dataMap.entrySet()) {
 try {
 Datum instancesDatum = entry.getValue();
 if (instancesDatum.value instanceof Instances) {
 count += ((Instances) instancesDatum.value).getInstanceList().size();
 }
 } catch (Exception ignore) {
 }
 }
 return count;
 }

 public Map<String, Datum> getDataMap() {
 return dataMap;
 }
}

1)添加任务

单机数据同步，首先要添加任务。服务注册会请求InstanceController post方法注册服务数据。

/**
* Register new instance.
* 服务数据注册
* @param request http request
* @return 'ok' if success
* @throws Exception any error during register
*/
@CanDistro //Distro协议(数据临时一致性协议)
@PostMapping
@Secured(parser = NamingResourceParser.class, action = ActionTypes.WRITE)
public String register(HttpServletRequest request) throws Exception {

    final String namespaceId = WebUtils
            .optional(request, CommonParams.NAMESPACE_ID, Constants.DEFAULT_NAMESPACE_ID);
    final String serviceName = WebUtils.required(request, CommonParams.SERVICE_NAME);
    NamingUtils.checkServiceNameFormat(serviceName);

    final Instance instance = parseInstance(request);
    // 注册服务实例
    serviceManager.registerInstance(namespaceId, serviceName, instance);
    return "ok";
}

注册服务数据

public void registerInstance(String namespaceId, String serviceName, Instance instance) throws NacosException {

    createEmptyService(namespaceId, serviceName, instance.isEphemeral());

    Service service = getService(namespaceId, serviceName);

    if (service == null) {
        throw new NacosException(NacosException.INVALID_PARAM,
                "service not found, namespace: " + namespaceId + ", service: " + serviceName);
    }
    // 添加服务实例
    addInstance(namespaceId, serviceName, instance.isEphemeral(), instance);
}

添加服务实例

/**
* Add instance to service.
* 将实例添加到服务
* @param namespaceId namespace
* @param serviceName service name
* @param ephemeral whether instance is ephemeral
* @param ips instances
* @throws NacosException nacos exception
*/
public void addInstance(String namespaceId, String serviceName, boolean ephemeral, Instance... ips)
 throws NacosException {

 String key = KeyBuilder.buildInstanceListKey(namespaceId, serviceName, ephemeral);

 Service service = getService(namespaceId, serviceName);
 // 同步防止高并发这种同步锁效率低
 synchronized (service) {
 List<Instance> instanceList = addIpAddresses(service, ephemeral, ips);

 Instances instances = new Instances();
 instances.setInstanceList(instanceList);

 consistencyService.put(key, instances);
 }
}

Put到中DistroConsistencyServiceImpl执行数据同步。

@Override
public void put(String key, Record value) throws NacosException {
    onPut(key, value);
    // 数据同步
    distroProtocol.sync(new DistroKey(key, KeyBuilder.INSTANCE_LIST_KEY_PREFIX), DataOperation.CHANGE,
            globalConfig.getTaskDispatchPeriod() / 2);
}

同步数据到远端的服务

/**
* Start to sync data to all remote server.
* 同步远端服务数据
* @param distroKey distro key of sync data
* @param action the action of data operation
*/
public void sync(DistroKey distroKey, DataOperation action, long delay) {

// 向除了自己以外的所有节点广播
    for (Member each : memberManager.allMembersWithoutSelf()) {
        DistroKey distroKeyWithTarget = new DistroKey(distroKey.getResourceKey(), distroKey.getResourceType(),
                each.getAddress());
        // 创建 Distro定时任务
        DistroDelayTask distroDelayTask = new DistroDelayTask(distroKeyWithTarget, action, delay);
        // 添加定时任务
        distroTaskEngineHolder
            // 获取延迟执行引擎
            .getDelayTaskExecuteEngine()
            // 添加延迟任务
            .addTask(distroKeyWithTarget, distroDelayTask);
        if (Loggers.DISTRO.isDebugEnabled()) {
            Loggers.DISTRO.debug("[DISTRO-SCHEDULE] {} to {}", distroKey, each.getAddress());
        }
    }
}

创建 Distro定时任务，可以看到里面有一些定时任务信息。

public class DistroDelayTask extends AbstractDelayTask {

 private final DistroKey distroKey;

 private DataOperation action;

 private long createTime;

 public DistroDelayTask(DistroKey distroKey, long delayTime) {
 this(distroKey, DataOperation.CHANGE, delayTime);
 }

 public DistroDelayTask(DistroKey distroKey, DataOperation action, long delayTime) {
 this.distroKey = distroKey;
 this.action = action;
 this.createTime = System.currentTimeMillis();
 setLastProcessTime(createTime);
 setTaskInterval(delayTime);
 }

 public DistroKey getDistroKey() {
 return distroKey;
 }

 public DataOperation getAction() {
 return action;
 }

 public long getCreateTime() {
 return createTime;
 }

 @Override
 public void merge(AbstractDelayTask task) {
 if (!(task instanceof DistroDelayTask)) {
 return;
 }
 DistroDelayTask newTask = (DistroDelayTask) task;
 if (!action.equals(newTask.getAction()) && createTime < newTask.getCreateTime()) {
 action = newTask.getAction();
 createTime = newTask.getCreateTime();
 }
 setLastProcessTime(newTask.getLastProcessTime());
 }
}

添加延迟任务，将任务存储到task(ConcurrentHashMap<Object, AbstractDelayTask>)中，后面执行任务的时候会从这个task中获取任务。

@Override
public void addTask(Object key, AbstractDelayTask newTask) {
    // 同步锁防止重复添加
    lock.lock();
    try {
        AbstractDelayTask existTask = tasks.get(key);
        if (null != existTask) {
            newTask.merge(existTask);
        }
        // 任务存储任务添加结束
        tasks.put(key, newTask);
    } finally {
        lock.unlock();
    }
}

2)任务执行

添加任务前DistroTaskEngineHolder会添加定时任务获取延迟执行引擎。

临时数据同步引擎，延迟执行任务。创建任务处理器并指定其处理任务。

/**
* Distro task engine holder.
* Doistro任务执行者
* 临时数据同步引擎延迟执行
* @author xiweng.yy
*/
@Component
public class DistroTaskEngineHolder {
    // 延迟执行引擎
    private final DistroDelayTaskExecuteEngine delayTaskExecuteEngine = new DistroDelayTaskExecuteEngine();

    private final DistroExecuteTaskExecuteEngine executeWorkersManager = new DistroExecuteTaskExecuteEngine();

    public DistroTaskEngineHolder(DistroComponentHolder distroComponentHolder) {
        // 任务处理器
        DistroDelayTaskProcessor defaultDelayTaskProcessor = new DistroDelayTaskProcessor(this, distroComponentHolder);
        // 指定任务处理器
        delayTaskExecuteEngine.setDefaultTaskProcessor(defaultDelayTaskProcessor);
    }

    public DistroDelayTaskExecuteEngine getDelayTaskExecuteEngine() {
        return delayTaskExecuteEngine;
    }

    public DistroExecuteTaskExecuteEngine getExecuteWorkersManager() {
        return executeWorkersManager;
    }

    public void registerNacosTaskProcessor(Object key, NacosTaskProcessor nacosTaskProcessor) {
        this.delayTaskExecuteEngine.addProcessor(key, nacosTaskProcessor);
    }
}

跟踪延迟执行引擎创建，可以看到其父类NacosDelayTaskExecuteEngine创建了延迟执行引擎。

/**
* 创建任务执行器
*/
public NacosDelayTaskExecuteEngine(String name, int initCapacity, Logger logger, long processInterval) {
 super(logger);
 tasks = new ConcurrentHashMap<Object, AbstractDelayTask>(initCapacity);
 processingExecutor = ExecutorFactory.newSingleScheduledExecutorService(new NameThreadFactory(name));
 processingExecutor
 // 定时执行ProcessRunnable任务
 .scheduleWithFixedDelay(new ProcessRunnable(), processInterval, processInterval, TimeUnit.MILLISECONDS);
}

定时执行器中创建了一个执行线程，会定时去执行这个线程ProcessRunnable

/**
* 定时任务执行的线程
*/
private class ProcessRunnable implements Runnable {

    @Override
    public void run() {
        try {
            processTasks();
        } catch (Throwable e) {
            getEngineLog().error(e.toString(), e);
        }
    }
}

线程回调NacosDelayTaskExecuteEngine中的processTasks方法

/**
* process tasks in execute engine.
*/
protected void processTasks() {
 Collection<Object> keys = getAllTaskKeys();
 for (Object taskKey : keys) {
 // 从task中获取任务这个任务在任务添加时已经添加到了task中
 AbstractDelayTask task = removeTask(taskKey);
 if (null == task) {
 continue;
 }
 // 获取任务执行对象 DistroDelayTaskProcessor
 // 先从缓存中获取缓存中没有就获取前面创建的对象这个对象在DistroTaskEngineHolder创建的时候已经被创建了
 NacosTaskProcessor processor = getProcessor(taskKey);
 if (null == processor) {
 getEngineLog().error("processor not found for task, so discarded. " + task);
 continue;
 }
 try {
 // ReAdd task if process failed
 if (!processor.process(task)) {
 // 存储任务
 retryFailedTask(taskKey, task);
 }
 } catch (Throwable e) {
 getEngineLog().error("Nacos task execute error : " + e.toString(), e);
 retryFailedTask(taskKey, task);
 }
 }
}

任务执行成功后会将任务存储到服务器缓存中。

private void retryFailedTask(Object key, AbstractDelayTask task) {
    // 设置任务进程时间
    task.setLastProcessTime(System.currentTimeMillis());
    // 存储任务
    addTask(key, task);
}

@Override
public void addTask(Object key, AbstractDelayTask newTask) {
    // 同步锁防止重复添加
    lock.lock();
    try {
        // 先获取任务
        AbstractDelayTask existTask = tasks.get(key);
        // 如果任务不为空合并更新任务
        if (null != existTask) {
            newTask.merge(existTask);
        }
        // 任务存储任务添加结束
        tasks.put(key, newTask);
    } finally {
        lock.unlock();
    }
}

前面做了3件事，获取任务、任务执行对象、执行任务。

获取任务

/**
* 获取一个任务并将任务从tasks中移除
*/
@Override
public AbstractDelayTask removeTask(Object key) {
    lock.lock();
    try {
        AbstractDelayTask task = tasks.get(key);
        if (null != task && task.shouldProcess()) {
            return tasks.remove(key);
        } else {
            return null;
        }
    } finally {
        lock.unlock();
    }
}

获取任务执行对象

private final ConcurrentHashMap<Object, NacosTaskProcessor> taskProcessors = new ConcurrentHashMap<Object, NacosTaskProcessor>();

@Override
public NacosTaskProcessor getProcessor(Object key) {
return taskProcessors.containsKey(key) ? taskProcessors.get(key) : defaultTaskProcessor;
}

DistroDelayTaskProcessor执行任务

/**
* 任务执行方法
*/
@Override
public boolean process(NacosTask task) {
    if (!(task instanceof DistroDelayTask)) {
        return true;
    }
    DistroDelayTask distroDelayTask = (DistroDelayTask) task;
    DistroKey distroKey = distroDelayTask.getDistroKey();
    if (DataOperation.CHANGE.equals(distroDelayTask.getAction())) {
        // 创建数据同步快照 后面回调用它的run方法
        DistroSyncChangeTask syncChangeTask = new DistroSyncChangeTask(distroKey, distroComponentHolder);
        // 同步快照任务
        distroTaskEngineHolder.getExecuteWorkersManager().addTask(distroKey, syncChangeTask);
        return true;
    }
    return false;
}

同步任务继续执行同步

/**
* 执行同步任务
*/
@Override
public void addTask(Object tag, AbstractExecuteTask task) {
    // 获取处理对象
    NacosTaskProcessor processor = getProcessor(tag);
    if (null != processor) {
        // 处理任务
        processor.process(task);
        return;
    }
    // 获取任务
    TaskExecuteWorker worker = getWorker(tag);
    // 添加任务
    worker.process(task);
}

@Override
public boolean process(NacosTask task) {
    if (task instanceof AbstractExecuteTask) {
        // 添加任务
        putTask((Runnable) task);
    }
    return true;
}

/**
* 获取任务执行器
*/
private TaskExecuteWorker getWorker(Object tag) {
    int idx = (tag.hashCode() & Integer.MAX_VALUE) % workersCount();
    // 任务执行器在创建DistroTaskEngineHolder对象时创建DistroExecuteTaskExecuteEngine时被创建
    return executeWorkers[idx];
}

将任务添加到队列中

/**
* 将同步任务对象添加到队列(BlockingQueue)中
*/
private void putTask(Runnable task) {
    try {
        // 添加DistroSyncChangeTask(前面创建的)到队列中
        queue.put(task);
    } catch (InterruptedException ire) {
        log.error(ire.toString(), ire);
    }
}

在前面创建DistroTaskEngineHolder对象时，会初始化DistroExecuteTaskExecuteEngine，初始化时会创建其父类TaskExecuteWorker。

public TaskExecuteWorker(final String name, final int mod, final int total, final Logger logger) {
 this.name = name + "_" + mod + "%" + total;
 this.queue = new ArrayBlockingQueue<Runnable>(QUEUE_CAPACITY);
 this.closed = new AtomicBoolean(false);
 this.log = null == logger ? LoggerFactory.getLogger(TaskExecuteWorker.class) : logger;
 new InnerWorker(name).start();
}

这里会创建一个线程，此时从队列中取出前面放进队列中的同步任务对象，去执行任务对象DistroSyncChangeTask数据同步对象的run方法。

/**
* Inner execute worker.
*/
private class InnerWorker extends Thread {

    InnerWorker(String name) {
        setDaemon(false);
        setName(name);
    }

    @Override
    public void run() {
        while (!closed.get()) {
            try {
                // 从队列取出任务
                Runnable task = queue.take();
                long begin = System.currentTimeMillis();
                // 执行任务对象 DistroSyncChangeTask 数据同步对象
                task.run();
                long duration = System.currentTimeMillis() - begin;
                if (duration > 1000L) {
                    log.warn("distro task {} takes {}ms", task, duration);
                }
            } catch (Throwable e) {
                log.error("[DISTRO-FAILED] " + e.toString(), e);
            }
        }
    }
}

DistroSyncChangeTask是一个线程，由InnerWorker 调用它的run方法，执行远程调用同步数据。

@Override
public void run() {
    Loggers.DISTRO.info("[DISTRO-START] {}", toString());
    try {
        String type = getDistroKey().getResourceType();
        // 查找数据
        DistroData distroData = distroComponentHolder.findDataStorage(type).getDistroData(getDistroKey());
        distroData.setType(DataOperation.CHANGE);
        // 向其他节点同步数据
        boolean result = distroComponentHolder.findTransportAgent(type).syncData(distroData, getDistroKey().getTargetServer());
        if (!result) {
            handleFailedTask();
        }
        Loggers.DISTRO.info("[DISTRO-END] {} result: {}", toString(), result);
    } catch (Exception e) {
        Loggers.DISTRO.warn("[DISTRO] Sync data change failed.", e);
        handleFailedTask();
    }
}

同步数据

/**
* 数据同步
*/
@Override
public boolean syncData(DistroData data, String targetServer) {
    if (!memberManager.hasMember(targetServer)) {
        return true;
    }
    byte[] dataContent = data.getContent();
    // 远程同步请求
    return NamingProxy.syncData(dataContent, data.getDistroKey().getTargetServer());
}

代理对象远程同步数据

/**
* Synchronize datum to target server.
* 同步数据到目标服务器
*
* @param data datum
* @param curServer target server address
* @return true if sync successfully, otherwise false
*/
public static boolean syncData(byte[] data, String curServer) {
 Map<String, String> headers = new HashMap<>(128);

 headers.put(HttpHeaderConsts.CLIENT_VERSION_HEADER, VersionUtils.version);
 headers.put(HttpHeaderConsts.USER_AGENT_HEADER, UtilsAndCommons.SERVER_VERSION);
 headers.put(HttpHeaderConsts.ACCEPT_ENCODING, "gzip,deflate,sdch");
 headers.put(HttpHeaderConsts.CONNECTION, "Keep-Alive");
 headers.put(HttpHeaderConsts.CONTENT_ENCODING, "gzip");

 try {
 // http发送同步数据请求 /v1/ns/distro/datum
 RestResult<String> result = HttpClient.httpPutLarge(
 "http://" + curServer + EnvUtil.getContextPath() + UtilsAndCommons.NACOS_NAMING_CONTEXT
 + DATA_ON_SYNC_URL, headers, data);
 if (result.ok()) {
 return true;
 }
 if (HttpURLConnection.HTTP_NOT_MODIFIED == result.getCode()) {
 return true;
 }
 throw new IOException("failed to req API:" + "http://" + curServer + EnvUtil.getContextPath()
 + UtilsAndCommons.NACOS_NAMING_CONTEXT + DATA_ON_SYNC_URL + ". code:" + result.getCode() + " msg: "
 + result.getData());
 } catch (Exception e) {
 Loggers.SRV_LOG.warn("NamingProxy", e);
 }
 return false;
}

Http发送同步数据请求 /v1/ns/distro/datum调用DistroController开始同步数据。

/**
* Synchronize datum.
* 数据同步(临时)
* @param dataMap data map
* @return 'ok' if success
* @throws Exception if failed
*/
@PutMapping("/datum")
public ResponseEntity onSyncDatum(@RequestBody Map<String, Datum<Instances>> dataMap) throws Exception {

 if (dataMap.isEmpty()) {
 Loggers.DISTRO.error("[onSync] receive empty entity!");
 throw new NacosException(NacosException.INVALID_PARAM, "receive empty entity!");
 }

 for (Map.Entry<String, Datum<Instances>> entry : dataMap.entrySet()) {
 if (KeyBuilder.matchEphemeralInstanceListKey(entry.getKey())) {
 String namespaceId = KeyBuilder.getNamespace(entry.getKey());
 String serviceName = KeyBuilder.getServiceName(entry.getKey());
 if (!serviceManager.containService(namespaceId, serviceName) && switchDomain
 .isDefaultInstanceEphemeral()) {
 serviceManager.createEmptyService(namespaceId, serviceName, true);
 }
 DistroHttpData distroHttpData = new DistroHttpData(createDistroKey(entry.getKey()), entry.getValue());
 distroProtocol.onReceive(distroHttpData);
 }
 }
 return ResponseEntity.ok("ok");
}

接收同步的发布数据，找到处理器处理。

/**
* Receive synced distro data, find processor to process.
* 接收同步的发布数据，找到处理器处理
* @param distroData Received data
* @return true if handle receive data successfully, otherwise false
*/
public boolean onReceive(DistroData distroData) {
    String resourceType = distroData.getDistroKey().getResourceType();
    DistroDataProcessor dataProcessor = distroComponentHolder.findDataProcessor(resourceType);
    if (null == dataProcessor) {
        Loggers.DISTRO.warn("[DISTRO] Can't find data process for received data {}", resourceType);
        return false;
    }
    return dataProcessor.processData(distroData);
}

数据处理

/**
* 数据处理
*/
@Override
public boolean processData(DistroData distroData) {
 DistroHttpData distroHttpData = (DistroHttpData) distroData;
 Datum<Instances> datum = (Datum<Instances>) distroHttpData.getDeserializedContent();
 // 数据同步将数据添加到dataStore中
 onPut(datum.key, datum.value);
 return true;
}

数据同步将数据添加到dataStore中

/**
* Put a new record.
*
* @param key key of record
* @param value record
*/
public void onPut(String key, Record value) {

 if (KeyBuilder.matchEphemeralInstanceListKey(key)) {
 Datum<Instances> datum = new Datum<>();
 datum.value = (Instances) value;
 datum.key = key;
 datum.timestamp.incrementAndGet();
 dataStore.put(key, datum);
 }

 if (!listeners.containsKey(key)) {
 return;
 }

 notifier.addTask(key, DataOperation.CHANGE);
}

Distro临时数据一致性数据存储结构，添加进这个数据模型中。

public void put(String key, Datum value) {
// key：Group@服务名称
dataMap.put(key, value);
}

2 .0.Sentinel Dashboard数据持久化

Sentinel的理念是开发者只需要关注资源的定义，当资源定义成功后可以动态增加各种流控降级规则。

Sentinel 提供两种方式修改规则：

• 通过API直接修改( loadRules )

• 通过DataSource适配不同数据源修改

手动通过API修改比较直观，可以通过以下几个API修改不同的规则：

FlowRuleManager.loadRules(List<FlowRule> rules); // 修改流控规则

DegradeRuleManager.loadRules(List<DegradeRule> rules); // 修改降级规则

手动修改规则(硬编码方式)一般仅用于测试和演示，生产上一般通过动态规则源的方式来动态管理规则。

2.1.动态配置原理

loadRules()方法只接受内存态的规则对象，但更多时候规则存储在文件、数据库或者配置中心当中。DataSource接口给我们提供了对接任意配置源的能力。相比直接通过API修改规则，实现DataSource接口是更加可靠的做法。

我们推荐通过控制台设置规则后将规则推送到统一的规则中心，客户端实现ReadableDataSource接口端监听规则中心实时获取变更，流程如下：

DataSource扩展常见的实现方式有:

• 拉模式：客户端主动向某个规则管理中心定期轮询拉取规则，这个规则中心可以是RDBMS、文件，甚至是VCS等。这样做的方式是简单，缺点是无法及时获取变更；

• 推模式：规则中心统一推送，客户端通过注册监听器的方式时刻监听变化，比如使用 Nacos、Zookeeper等配置中心。这种方式有更好的实时性和一致性保证。

Sentinel目前支持以下数据源扩展：

• Pull-based: 动态文件数据源、Consul, Eureka

• Push-based: ZooKeeper, Redis, Nacos, Apollo, etcd

2.2.Sentinel+Nacos数据持久化

我们要想实现Sentinel+Nacos数据持久化，需要下载Sentinel控制台源码，关于源码下载我们这里就不再重复了。

在Sentinel Dashboard中配置规则之后重启应用就会丢失，所以实际生产环境中需要配置规则的持久化实现，Sentinel提供多种不同的数据源来持久化规则配置，包括file，redis、nacos、zk。

这就需要涉及到Sentinel Dashboard的规则管理及推送功能：集中管理和推送规则。sentinel-core提供 API 和扩展接口来接收信息。开发者需要根据自己的环境，选取一个可靠的推送规则方式；同时，规则最好在控制台中集中管理。

我们采用Push模式，即Sentinel-Dashboard统一管理配置，然后将规则统一推送到Nacos并持久化(生成配置文件)，最后客户端监听Nacos(这一部了解使用过Nacos的话应该很熟，采用ConfifigService.getConfg()方法获取配置文件)，下发配置生成Rule。

这张图的意思我们解释说明一下：

1:Sentinel Dashboard界面配置流控规则---发布/推送--->Nacos生成配置文件并持久化；

2:通过Nacos配置文件修改流控规则---拉取--->Sentinel Dashboard界面显示最新的流控规则。

在Nacos控制台上修改流控制，虽然可以同步到Sentinel Dashboard，但是Nacos此时应该作为一个流控规则的持久化平台，所以正常操作过程应该是开发者在Sentinel Dashboard上修改流控规则后同步到Nacos，遗憾的是目前Sentinel Dashboard不支持该功能。

如果公司没有统一在Sentinel Dashboard或Nacos中二选一进行配置，而是一会在Sentinel Dashboard配置，一会在Nacos配置。那么就会出现很严重的问题(流控规则达不到预期，配置数据不一致)，所以推荐使用Sentinel Dashboard统一界面进行配置管理流控规则。

我们接下来基于Sentinel1.8.1开始改造Sentinel Dashboard，使他能结合Nacos实现数据持久化。

2.2.1.Dashboard改造分析

Sentinel Dashboard的流控规则下的所有操作，都会调用Sentinel-Dashboard源码中的FlowControllerV1类，这个类中包含流控规则本地化的CRUD操作；

在com.alibaba.csp.sentinel.dashboard.controller.v2包下存在一个FlowControllerV2；类，这个类同样提供流控规则的CURD，与V1不同的是，它可以实现指定数据源的规则拉取和发布。

上面代码就是FlowControllerV2 部分代码，分别实现了拉取规则和推送规则：

1:DynamicRuleProvider：动态规则的拉取，从指定数据源中获取控制后在Sentinel Dashboard中展示。

2:DynamicRulePublisher：动态规则发布，将在Sentinel Dashboard中修改的规则同步到指定数据源中。

我们只需要扩展这两个类，然后集成Nacos来实现Sentinel Dashboard规则同步。

2.2.2.页面改造

在目录resources/app/scripts/directives/sidebar找到sidebar.html，里面有关于V1版本的请求入口：

流控规则</a>

</li>

对应的JS(app.js)请求如下，可以看到请求就是V1版本的Controller，那么之后的改造需要重新对应V2版本的Controller。

.state('dashboard.flowV1', {

templateUrl: 'app/views/flow_v1.html',

url: '/flow/:app',

controller: 'FlowControllerV1',

resolve: {

loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) {

return $ocLazyLoad.load({

name: 'sentinelDashboardApp',

files: [

'app/scripts/controllers/flow_v1.js',

]

});

}]

}

})

2.2.3.Nacos配置

在源码中虽然官方提供了test示例(即test目录)下关于Nacos等持久化示例，但是具体的实现还需要一些细节，比如在Sentinel Dashboard配置Nacos的serverAddr、namespace、groupId，并且通过Nacos获取配置文件获取服务列表等。

我们可以打开NacosConfig源码，NacosConfig中ConfigFactory.createConfigService("localhost") 并没有实现创建具体的nacos config service，而是默认 localhost ,application.properties文件中也没有Nacos的相关配置，这些都需要我们额外配置，NacosConfig代码如下：

@Configuration
public class NacosConfig {
 @Bean
 public Converter<List<FlowRuleEntity>, String> flowRuleEntityEncoder() {
 return JSON::toJSONString;
 }
 @Bean
 public Converter<String, List<FlowRuleEntity>> flowRuleEntityDecoder() {
 return s -> JSON.parseArray(s, FlowRuleEntity.class);
 }
 @Bean
 public ConfigService nacosConfigService() throws Exception {
 return ConfigFactory.createConfigService("localhost");
 }
}

如果我们需要把数据存储到Nacos，在NacosConfigUtils已经指定了默认的流控规则配置文件的groupId等，如果需要指定的话这里也需要修改.

public final class NacosConfigUtil {
    //Nacos中对应的GroupID
    public static final String GROUP_ID = "SENTINEL_GROUP";
    //文件后半部分
    public static final String FLOW_DATA_ID_POSTFIX = "-flow-rules";
    public static final String PARAM_FLOW_DATA_ID_POSTFIX = "-param-rules";
    public static final String CLUSTER_MAP_DATA_ID_POSTFIX = "-cluster-map";
//略...
}

我们在application.properties中配置Nacos：

spring.cloud.sentinel.datasource.flow.nacos.server-addr=nacos:8848

spring.cloud.sentinel.datasource.flow.nacos.data-id=${spring.application.name}-flow-rules

spring.cloud.sentinel.datasource.flow.nacos.group-id=SENTINEL_GROUP

spring.cloud.sentinel.datasource.flow.nacos.data-type=json

spring.cloud.sentinel.datasource.flow.nacos.rule-type=flow

2.3.4.Dashboard持久化改造

我们接下来开始改造Dashboard源码，官方提供的Nacos持久化用例都是在test目录下，所以scope需要去除test，需要sentinel-datasource-nacos包的支持。之后将修改好的源码放在源码主目录com.alibaba.csp.sentinel.dashboard下，而不是继续在test目录下。

<groupId>com.alibaba.csp</groupId>

<artifactId>sentinel-datasource-nacos</artifactId>

</dependency>

找到resources/app/scripts/directives/sidebar/sidebar.html文件修改，修改flflowV1为flflow，去掉V1，这样的话会调用FlowControllerV2接口

修改前：

流控规则</a>

</li>

修改后：

流控规则</a>

</li>

这样就可以通过js跳转至FlowControllerV2了,app.js代码如下：

.state('dashboard.flow', {

templateUrl: 'app/views/flow_v2.html',

url: '/v2/flow/:app',

controller: 'FlowControllerV2',

resolve: {

loadMyFiles: ['$ocLazyLoad', function ($ocLazyLoad) {

return $ocLazyLoad.load({

name: 'sentinelDashboardApp',

files: [

'app/scripts/controllers/flow_v2.js',

]

});

}]

}

})

2.3.5.Nacos配置创建

我们采用官方的约束，即默认Nacos适配的dataId和groupId约定,所以不需要修改NacosConfigUtil.java了,配置如下：

groupId: SENTINEL_GROUP

流控规则 dataId: {appName}-flow-rules，比如应用名为 appA，则 dataId 为 appA-flow- rules

我们在 application.properties 中配置Nacos服务信息：

# nacos config server

sentinel.nacos.serverAddr=nacos:8848

sentinel.nacos.namespace=

sentinel.nacos.group-id=SENTINEL_GROUP

sentinel.nacos.password=nacos

sentinel.nacos.username=nacos

接下来创建读取nacos配置的NacosPropertiesConfiguration文件并且 application.properties指定配置。

@ConfigurationProperties(prefix = "sentinel.nacos")

public class NacosPropertiesConfiguration {

private String serverAddr;

private String dataId;

private String groupId = "SENTINEL_GROUP"; // 默认分组

private String namespace;

private String username;

private String password;

//get set

}

2.3.6.改造源码

2.3.6.1.改造NacosConfifig

我们最后改造NacosConfig，让NacosConfig做两件事：

1)注入Convert转换器，将FlowRuleEntity转化成FlowRule，以及反向转化

2)注入Nacos配置服务ConfifigService

@EnableConfigurationProperties(NacosPropertiesConfiguration.class)
@Configuration
public class NacosConfig {
 @Bean
 public Converter<List<FlowRuleEntity>, String> flowRuleEntityEncoder() {
 return JSON::toJSONString;
 }

 @Bean
 public Converter<String, List<FlowRuleEntity>> flowRuleEntityDecoder() {
 return s -> JSON.parseArray(s, FlowRuleEntity.class);
 }

 @Bean
 public ConfigService nacosConfigService(NacosPropertiesConfiguration nacosPropertiesConfiguration) throws Exception {
 Properties properties = new Properties();
 properties.put(PropertyKeyConst.SERVER_ADDR,
 nacosPropertiesConfiguration.getServerAddr());
 properties.put(PropertyKeyConst.NAMESPACE,
 nacosPropertiesConfiguration.getNamespace());
 properties.put(PropertyKeyConst.USERNAME,
 nacosPropertiesConfiguration.getUwername());
 properties.put(PropertyKeyConst.PASSWORD,
 nacosPropertiesConfiguration.getPassword());
 return ConfigFactory.createConfigService(properties);
 // return ConfigFactory.createConfigService("localhost");
 }
}

2.3.6.2.动态获取流控规则

动态实现从Nacos配置中心获取流控规则需要重写FlowRuleNacosProvider与FlowRuleNacosPublisher类。

1)重写FlowRuleNacosProvider类

@Service("flowRuleNacosProvider")
public class FlowRuleNacosProvider implements DynamicRuleProvider<List<FlowRuleEntity>> {
 public static final Logger log = LoggerFactory.getLogger(FlowRuleNacosProvider.class);
 @Autowired
 private ConfigService configService;
 @Autowired
 private Converter<String, List<FlowRuleEntity>> converter;
 /**
 * 1)通过ConfigService的getConfig()方法从Nacos Config Server读取指定配置信息
 * 2)通过转为converter转化为FlowRule规则
 * @param appName
 * @return
 * @throws Exception
 */
 @Override
 public List<FlowRuleEntity> getRules(String appName) throws Exception {
 String rules = configService.getConfig(appName + NacosConfigUtil.FLOW_DATA_ID_POSTFIX, NacosConfigUtil.GROUP_ID, 3000);
 log.info("obtain flow rules from nacos config:{}", rules);
 if (StringUtil.isEmpty(rules)) {
 return new ArrayList<>();
 }
 return converter.convert(rules);
 }
}

2)重写FlowRuleNacosPublisher类：

@Service("flowRuleNacosPublisher")
public class FlowRuleNacosPublisher implements DynamicRulePublisher<List<FlowRuleEntity>> {
 public static final Logger log = LoggerFactory.getLogger(FlowRuleNacosPublisher.class);
 @Autowired
 private ConfigService configService;
 @Autowired
 private Converter<List<FlowRuleEntity>, String> converter;
 /**
 * 通过configService的publishConfig()方法将rules发布到nacos
 * @param app app name
 * @param rules list of rules to push
 * @throws Exception
 */
 @Override
 public void publish(String app, List<FlowRuleEntity> rules) throws Exception {
 AssertUtil.notEmpty(app, "app name cannot be empty");
 if (rules == null) {
 return;
 }
 log.info("sentinel dashboard push rules: {}", rules);
 configService.publishConfig(app + NacosConfigUtil.FLOW_DATA_ID_POSTFIX, NacosConfigUtil.GROUP_ID, converter.convert(rules));
 }
}

3)替换默认对象

修改FlowControllerV2类，使用@Qulififier将上面配置的两个类注入进来

2.3.6.3.数据持久化测试

我们接下来将程序打包，如下图：

打包好程序后，再将程序运行起来：

java -Dserver.port=8858 -Dcsp.sentinel.dashboard.server=localhost:8858 -Dproject.name=sentinel-dashboard -jar sentinel-dashboard.jar

运行起来后，我们可以发现在Nacos中多了一个服务，如下图：

我们随意增加一个流控规则，如下图：

点击新增之后，我们可以发现SENTILE_GROUP组下多了一个文件，sentinel-flow-rule文件，效果如下：

2.3.系统规则定义

上面改造样子使用的限流规则，还有其他一些规则也可以去配置。

我们来配置一个系统服务规则：

在NacosConfigUtil中新增配置定义为系统规则配置"-system-rules"

public final class NacosConfigUtil {

    public static final String GROUP_ID = "SENTINEL_GROUP";

    public static final String FLOW_DATA_ID_POSTFIX = "-flow-rules";
    public static final String PARAM_FLOW_DATA_ID_POSTFIX = "-param-rules";
    public static final String CLUSTER_MAP_DATA_ID_POSTFIX = "-cluster-map";

    // 系统规则
    public static final String SYSTEM_DATA_ID_POSTFIX = "-system-rules";
    /**
     * cc for `cluster-client`
     */
    public static final String CLIENT_CONFIG_DATA_ID_POSTFIX = "-cc-config";
    /**
     * cs for `cluster-server`
     */
    public static final String SERVER_TRANSPORT_CONFIG_DATA_ID_POSTFIX = "-cs-transport-config";
    public static final String SERVER_FLOW_CONFIG_DATA_ID_POSTFIX = "-cs-flow-config";
    public static final String SERVER_NAMESPACE_SET_DATA_ID_POSTFIX = "-cs-namespace-set";

    private NacosConfigUtil() {}
}

复制FlowRuleNacosProvider与FlowRuleNacosPublisher类规则推送，并修改为系统规则。

SystemRuleNacosProvider：

@Component("systemRuleNacosProvider")
public class SystemRuleNacosProvider implements DynamicRuleProvider<List<SystemRuleEntity>> {

 @Autowired
 private ConfigService configService;
 @Autowired
 private Converter<String, List<SystemRuleEntity>> converter;

 @Override
 public List<SystemRuleEntity> getRules(String appName) throws Exception {
 String rules = configService.getConfig(appName + NacosConfigUtil.SYSTEM_DATA_ID_POSTFIX,
 NacosConfigUtil.GROUP_ID, 3000);
 if (StringUtil.isEmpty(rules)) {
 return new ArrayList<>();
 }
 return converter.convert(rules);
 }
}

SystemRuleNacosPublisher：

@Component("systemRuleNacosPublisher")
public class SystemRuleNacosPublisher implements DynamicRulePublisher<List<SystemRuleEntity>> {

 @Autowired
 private ConfigService configService;
 @Autowired
 private Converter<List<SystemRuleEntity>, String> converter;

 @Override
 public void publish(String app, List<SystemRuleEntity> rules) throws Exception {
 AssertUtil.notEmpty(app, "app name cannot be empty");
 if (rules == null) {
 return;
 }
 configService.publishConfig(app + NacosConfigUtil.SYSTEM_DATA_ID_POSTFIX,
 NacosConfigUtil.GROUP_ID, converter.convert(rules));
 }
}

规则转换器：在NacosConfig中添加规则转换器，即数据发送给nacos和接收nacos信息需要处理一下。

@Configuration
public class NacosConfig {
 @Value("${nacos.address}")
 private String address;
 @Value("${nacos.namespace}")
 private String namespace;
 @Value("${nacos.username}")
 private String username;
 @Value("${nacos.password}")
 private String password;

 @Bean
 public ConfigService nacosConfigService() throws Exception {
 // nacos配置
 Properties properties = new Properties();
 properties.put(PropertyKeyConst.SERVER_ADDR, address);
 properties.put(PropertyKeyConst.NAMESPACE, namespace);
 properties.put(PropertyKeyConst.USERNAME, username);
 properties.put(PropertyKeyConst.PASSWORD, password);
 //return ConfigFactory.createConfigService("localhost");
 return ConfigFactory.createConfigService(properties);
 }

 /**
 * 编码 sentinel将数据发送给nacos要编码
 */
 @Bean
 public Converter<List<FlowRuleEntity>, String> flowRuleEntityEncoder() {
 return JSON::toJSONString;
 }

 /**
 * 解码 sentinel从nacos获取数据要解码
 */
 @Bean
 public Converter<String, List<FlowRuleEntity>> RuleEntityDecoder() {
 return s -> JSON.parseArray(s, FlowRuleEntity.class);
 }

 /**
 * 规则转换器推送将队则转成JSON字符串
 */
 @Bean
 public Converter<List<SystemRuleEntity>, String> systemRuleEntityEncoder() {
 return JSON::toJSONString;
 }

 /**
 * 规则转换器读取将nacos中的JSON规则转换成SystemRuleEntity
 */
 @Bean
 public Converter<String, List<SystemRuleEntity>> systemRuleEntityDecoder() {
 return s -> JSON.parseArray(s, SystemRuleEntity.class);
 }
}

修改SystemController

@RestController
@RequestMapping("/system")
public class SystemController {

 private final Logger logger = LoggerFactory.getLogger(SystemController.class);

 @Autowired
 private RuleRepository<SystemRuleEntity, Long> repository;
 @Autowired
 @Qualifier("systemRuleNacosProvider")
 private DynamicRuleProvider<List<SystemRuleEntity>> ruleProvider;
 @Autowired
 @Qualifier("systemRuleNacosPublisher")
 private DynamicRulePublisher<List<SystemRuleEntity>> rulePublisher;

 private <R> Result<R> checkBasicParams(String app, String ip, Integer port) {
 if (StringUtil.isEmpty(app)) {
 return Result.ofFail(-1, "app can't be null or empty");
 }
 if (port <= 0 || port > 65535) {
 return Result.ofFail(-1, "port should be in (0, 65535)");
 }
 return null;
 }

 @GetMapping("/rules.json")
 @AuthAction(PrivilegeType.READ_RULE)
 public Result<List<SystemRuleEntity>> apiQueryMachineRules(String app, String ip,
 Integer port) {
 Result<List<SystemRuleEntity>> checkResult = checkBasicParams(app, ip, port);
 if (checkResult != null) {
 return checkResult;
 }
 try {
 List<SystemRuleEntity> rules = ruleProvider.getRules(app);
 rules = repository.saveAll(rules);
 return Result.ofSuccess(rules);
 } catch (Throwable throwable) {
 logger.error("Query machine system rules error", throwable);
 return Result.ofThrowable(-1, throwable);
 }
 }

 private int countNotNullAndNotNegative(Number... values) {
 int notNullCount = 0;
 for (int i = 0; i < values.length; i++) {
 if (values[i] != null && values[i].doubleValue() >= 0) {
 notNullCount++;
 }
 }
 return notNullCount;
 }

 @RequestMapping("/new.json")
 @AuthAction(PrivilegeType.WRITE_RULE)
 public Result<SystemRuleEntity> apiAdd(String app, String ip, Integer port,
 Double highestSystemLoad, Double highestCpuUsage, Long avgRt,
 Long maxThread, Double qps) {

 Result<SystemRuleEntity> checkResult = checkBasicParams(app, ip, port);
 if (checkResult != null) {
 return checkResult;
 }

 int notNullCount = countNotNullAndNotNegative(highestSystemLoad, avgRt, maxThread, qps, highestCpuUsage);
 if (notNullCount != 1) {
 return Result.ofFail(-1, "only one of [highestSystemLoad, avgRt, maxThread, qps,highestCpuUsage] "
 + "value must be set > 0, but " + notNullCount + " values get");
 }
 if (null != highestCpuUsage && highestCpuUsage > 1) {
 return Result.ofFail(-1, "highestCpuUsage must between [0.0, 1.0]");
 }
 SystemRuleEntity entity = new SystemRuleEntity();
 entity.setApp(app.trim());
 entity.setIp(ip.trim());
 entity.setPort(port);
 // -1 is a fake value
 if (null != highestSystemLoad) {
 entity.setHighestSystemLoad(highestSystemLoad);
 } else {
 entity.setHighestSystemLoad(-1D);
 }

 if (null != highestCpuUsage) {
 entity.setHighestCpuUsage(highestCpuUsage);
 } else {
 entity.setHighestCpuUsage(-1D);
 }

 if (avgRt != null) {
 entity.setAvgRt(avgRt);
 } else {
 entity.setAvgRt(-1L);
 }
 if (maxThread != null) {
 entity.setMaxThread(maxThread);
 } else {
 entity.setMaxThread(-1L);
 }
 if (qps != null) {
 entity.setQps(qps);
 } else {
 entity.setQps(-1D);
 }
 Date date = new Date();
 entity.setGmtCreate(date);
 entity.setGmtModified(date);
 try {
 entity = repository.save(entity);
 } catch (Throwable throwable) {
 logger.error("Add SystemRule error", throwable);
 return Result.ofThrowable(-1, throwable);
 }
 publishRules(app);
 return Result.ofSuccess(entity);
 }

 @GetMapping("/save.json")
 @AuthAction(PrivilegeType.WRITE_RULE)
 public Result<SystemRuleEntity> apiUpdateIfNotNull(Long id, String app, Double highestSystemLoad,
 Double highestCpuUsage, Long avgRt, Long maxThread, Double qps) {
 if (id == null) {
 return Result.ofFail(-1, "id can't be null");
 }
 SystemRuleEntity entity = repository.findById(id);
 if (entity == null) {
 return Result.ofFail(-1, "id " + id + " dose not exist");
 }

 if (StringUtil.isNotBlank(app)) {
 entity.setApp(app.trim());
 }
 if (highestSystemLoad != null) {
 if (highestSystemLoad < 0) {
 return Result.ofFail(-1, "highestSystemLoad must >= 0");
 }
 entity.setHighestSystemLoad(highestSystemLoad);
 }
 if (highestCpuUsage != null) {
 if (highestCpuUsage < 0) {
 return Result.ofFail(-1, "highestCpuUsage must >= 0");
 }
 if (highestCpuUsage > 1) {
 return Result.ofFail(-1, "highestCpuUsage must <= 1");
 }
 entity.setHighestCpuUsage(highestCpuUsage);
 }
 if (avgRt != null) {
 if (avgRt < 0) {
 return Result.ofFail(-1, "avgRt must >= 0");
 }
 entity.setAvgRt(avgRt);
 }
 if (maxThread != null) {
 if (maxThread < 0) {
 return Result.ofFail(-1, "maxThread must >= 0");
 }
 entity.setMaxThread(maxThread);
 }
 if (qps != null) {
 if (qps < 0) {
 return Result.ofFail(-1, "qps must >= 0");
 }
 entity.setQps(qps);
 }
 Date date = new Date();
 entity.setGmtModified(date);
 try {
 entity = repository.save(entity);
 } catch (Throwable throwable) {
 logger.error("save error:", throwable);
 return Result.ofThrowable(-1, throwable);
 }
 publishRules(app);
 return Result.ofSuccess(entity);
 }

 @RequestMapping("/delete.json")
 @AuthAction(PrivilegeType.DELETE_RULE)
 public Result<?> delete(Long id) {
 if (id == null) {
 return Result.ofFail(-1, "id can't be null");
 }
 SystemRuleEntity oldEntity = repository.findById(id);
 if (oldEntity == null) {
 return Result.ofSuccess(null);
 }
 try {
 repository.delete(id);
 } catch (Throwable throwable) {
 logger.error("delete error:", throwable);
 return Result.ofThrowable(-1, throwable);
 }
 publishRules(oldEntity.getApp());
 return Result.ofSuccess(id);
 }

 private void publishRules(String app) {
 List<SystemRuleEntity> rules = repository.findAllByApp(app);
 try {
 rulePublisher.publish(app, rules);
 } catch (Exception e) {
 e.printStackTrace();
 }
 }
}

重启服务，系统规则就配置好了。

服务端拉取规则配置需要引入相关依赖包，配置相关sentinel配置。

在服务端引入相关依赖包。


<dependency>
 <groupId>com.alibaba.csp</groupId>
 <artifactId>sentinel-datasource-nacos</artifactId>
 <version>1.8.0</version>
</dependency>

配置sentinel支持nacos配置，从nacos中加载对应的配置规则。

spring:
  application:
    name: hailtaxi-driver
  cloud:
    sentinel:
      transport:
        port: 8719
        dashboard: 127.0.0.1:8080
      datasource:
        nacos:
          server-addr: 127.0.0.1:8848
          username: nacos
          password: nacos
          namespace: 1cc805fd-009f-4ca9-8ceb-f424c9b8babd
          groupId: DEFAULT_GROUP
          dataId: ${ spring.application.name}-flow-rules
          rule-type: flow
        system:
          nacos:
            server-addr: 127.0.0.1:8848
            username: nacos
            password: nacos
            namespace: 1cc805fd-009f-4ca9-8ceb-f424c9b8babd
            groupId: DEFAULT_GROUP
            dataId: ${ spring.application.name}-system-rules
            rule-type: system

启动服务，新增系统规则，配置CPU使用率。

访问测试接口：http://127.0.0.1:18888/driver/info/1

当CUP使用率低于10%，接口才能被调用。

当CPU使用率高于10%，系统将会被熔断，拒绝被访问。