background

Some time ago, there were news reports that foreign HashiCorp announced on its official website that it is not allowed to use, deploy and install the enterprise version products and software of the company in China.

Among them, Consul is a middleware of service discovery and configuration center that Java spring cloud developers are very familiar with. Many people worry about whether Consul will be affected. At present, HashiCorp only prohibits the use of the commercial version, and has not yet implemented the open source version. Restrictions, so those who use Consul don't have to worry. However, with the development of time, the confrontation in different regions will continue to escalate, and maybe one day the open source version will be announced to be banned, so we need to know how to replace Consul.

In 2008, when zk had not yet come out, Alibaba needed to do service discovery internally, so it developed ConfigServer. After ten years, in July 2018, Alibaba released Nacos (ConfigServer open source implementation) Version 0.1.0, it has been almost two years since version 1.3.0, and now it can support many functions:

Service registration and discovery: nacos has been integrated with many rpc frameworks, such as dubbo, springcloud, etc., which is convenient for us to use out of the box. At the same time, it also opens a relatively simple api for us to customize our own rpc.
Configuration management: a configuration management center similar to apllo, let us not need to write the configuration in the file, and carry out unified management in the background.
Address server: It is convenient for us to address nacos in different environments and different isolation scenarios.
Security and stability: performance monitoring, encrypted transmission, permission control management, etc.

For nacos, the biggest core functions are service registration and configuration management. My article mainly introduces these two modules. This article mainly introduces some uses, principles and comparisons related to Nacos service discovery-registration. optimization.

basic concept

First, let's take a look at some basic concepts of service discovery-registration in Nacos:

Namespace: Namespace belongs to the top-level structure of Nacos and is used for tenant-level isolation. We most commonly use different environments such as test environments and online environments to isolate them.
Service: The concept of service corresponds one-to-one with our usual microservices, such as order service, logistics service and so on. A namespace can have multiple services, and different namespaces can have the same service. For example, both the test environment and the online environment can have order services.
Virtual cluster: All machines under a service form a cluster. In Nacos, the cluster can be further divided into virtual clusters as needed.
Instance: A rough understanding is that a machine or a virtual machine is an instance, and a fine-grained understanding is a process with an accessible network address (IP:Port) for one or more services.

The above is the service domain model diagram given in the Nacos official website document. From the diagram, we can know that the hierarchical relationship belongs to: service-cluster-instance. At the same time, some data will be saved in the service, cluster and instance for other needs.

In fact, when it comes to service registration, many people first think of Zookeeper. In fact, ZK does not directly provide the function of service registration and subscription. To implement these functions in ZK, you must divide the file directory one by one yourself, which is very inconvenient, and The Api of ZK is also very difficult to use. For Nacos, the Api of service registration is used as follows:

        Properties properties = new Properties();
        properties.setProperty("serverAddr", System.getProperty("serverAddr"));
        properties.setProperty("namespace", System.getProperty("namespace"));

        NamingService naming = NamingFactory.createNamingService(properties);

        naming.registerInstance("microservice-mmp-marketing", "11.11.11.11", 8888, "TEST1");
        
        naming.subscribe("microservice-mmp-marketing", new EventListener() {
            @Override
            public void onEvent(Event event) {
                System.out.println(((NamingEvent)event).getServiceName());
                System.out.println(((NamingEvent)event).getInstances());
            }
        });

We only need to create a NamingService, and then call the registerInstance method and the subscribe method to complete the registration and subscription of our service.

If you are interested in Nacos quick access, you can go to the official website for a detailed look, and I will not introduce it here, https://nacos.io/zh-cn/docs/quick-start.html

AP or CP

CAP

When it comes to distributed systems, it must be inseparable from the CAP theorem, which is called Brewer's theorem. For architects designing distributed systems (not just distributed transactions), CAP is your starting point.

C (Consistency): For a given client, a read operation can return the latest write operation. For data distributed on different nodes, if the data is updated on a certain node, if the latest data can be read by other nodes, it is called strong consistency. If a node does not read the data If you get it, it is distributed inconsistency.
A (availability): A non-faulty node returns a reasonable response (not an error and timeout response) in a reasonable amount of time. The two keys to availability are reasonable time and reasonable response. Reasonable time means that the request cannot be blocked indefinitely and should be returned in a reasonable time. A reasonable response means that the system should explicitly return the result and the result is correct, where correct means that it should return 50 instead of 40, for example.
P (Partition Tolerance): The system can continue to work when a network partition occurs. For example, there are multiple machines in a cluster, and there is a problem with the network of one machine, but the cluster can still work normally.

Anyone who is familiar with CAP knows that the three cannot be shared. If you are interested, you can search for the proof of CAP. In a distributed system, the network cannot be 100% reliable, and partitioning is actually an inevitable phenomenon. If we choose CA and give up P, Then when a partition occurs, in order to ensure consistency, the request must be rejected at this time, but A does not allow it, so it is theoretically impossible for a distributed system to choose the CA architecture, only the CP or AP architecture.

For CP, to give up availability and pursue consistency and partition fault tolerance, our zookeeper is actually the pursuit of strong consistency.

For AP, giving up consistency (the consistency mentioned here is strong consistency) and pursuing partition fault tolerance and availability is the choice of many distributed systems when designing, and the latter BASE is also extended according to AP.

By the way, CAP theory ignores network latency, that is, when a transaction is committed, it is copied from node A to node B, but in reality this is obviously impossible, so there will always be some time inconsistency. At the same time, choosing two of the CAP, for example, you choose CP, does not tell you to give up A. Because the probability of P appearing is so small, you still need to guarantee CA most of the time. Even if the partition appears, you have to prepare for the later A, for example, by means of some logs, other machines will be restored to usable.

Choice of Registry

As mentioned above, all distributed systems will decide on CAP, and the same registry is no exception. Zookeeper is the first choice for many service registration centers. My current company is also using ZooKeeper as a registration center, but with the development of the company, ZK is becoming more and more unstable, and service discovery in multiple computer rooms is also very difficult. In this article Why does Alibaba not use ZooKeeper for service discovery in the article ? , explains in more detail why Ali does not use ZK as a registration center. Here I briefly explain:

Unsatisfactory performance, unable to scale horizontally: Students who are familiar with ZK know that ZK writes data to the leader (master node), so it is difficult to scale horizontally. When the company reaches a certain scale, ZK is not suitable as a registration center. Frequent reading and writing can easily lead to ZK instability. In this case, multiple ZK clusters can be divided, but there is a problem. At the beginning, each cluster may not deal with each other, but if there is any collaborative business in the later stage, the services of each cluster will call each other. a difficult point.
The ZooKeeper API is difficult to use: The use of Zookeeper really requires a relatively proficient expert. You need to be familiar with many of its exceptions and what to do with these exceptions.
The registry does not need to store some historical changes: theoretically, the registry only needs to know which services and instances are registered in the registry at this moment, but ZK will continue to record transaction logs for subsequent repairs.
The same computer room cannot be connected: If we have a three-computer room disaster recovery 5-node deployment structure, as shown in the following figure: If there is a network partition between computer room 3 and computer room 1 and 2, that is, the inside of the computer room is good, but there is no difference between the computer rooms. Connected, because the ZK of the computer room is partitioned, the current ZK will also be unavailable, then the services in the computer room will not be able to use the registration center and cannot call each other. Obviously, this is not allowed. If the network is good, calls should be allowed within the same computer room.

Based on the above, ZK is not suitable for our registration center. In other words, the registration center does not need CP, and we should provide more AP.

Protocols in Nacos

Distro

An ephemeral field is provided in the Instance (instance) of Nacos. This field is of type bool. The meaning of this field is similar to that in ZK. It represents whether it is a temporary node. In Nacos, if it is a temporary node, the AP protocol will be used. If it is not a temporary node, it will go to the CP. Of course, all instances in the registry are actually temporary nodes by default.

In order to implement AP in Nacos, a set of Distro protocol has been customized. Let's analyze what Distro is:

Pure memory save

All data in Distro is stored in memory, in DistroConsistencyService:

It can be seen that Distro uses ConcurrentHashMap as a storage container, and does not need to use additional files for storage. Some students will ask if my machine goes down and all memory information is lost, how can I recover this part of the data?

In the load method of DistroConsistencyService, we traverse all non-own servers, and then synchronize the data. If one of them is successfully synchronized, there is no need to send synchronization information to other servers. This will restore all data.

eventually consistent

Although we are talking about AP, we still need to ensure the final consistency to prevent data inconsistency of each node for a long time. In the Put method of DistroConsistencyService, a task will be added to the TaskDispatcher, as shown in the figure below: TaskDispatcher is actually called DataSyncTaskDispatcher, which is more appropriate and is mainly used for To synchronize data:

The core logic is the whie loop above, mainly looking at the code I marked in red, here is not to send one update when one comes, if the service changes frequently, if one sends one, the efficiency must be very low, so here we take an update. The merging strategy, if the updated data reaches a certain amount, the default here is 1000, or the last transmission has exceeded a certain time. The default here is 2s, and a transmission will be performed. The transmission here is also to generate a SyncTask and put it in the thread Send asynchronously to the pool.

At this time, some students will ask, what should I do if one of my machines has just been launched and I just haven't received the updated data? There will also be a bottom-up strategy TimedSync in Nacos. This task will be executed every 5s. The specific execution task code is as follows:

Pay attention to the part circled in red. This is not to synchronize all data, but to traverse all the data. Only the part of the data that belongs to your own management will be synchronized (what data belongs to your own management? I will discuss it in detail in the next section), and then get All non-own servers check and send these data.

Through these two methods: real-time update and timed update, we can ensure that the data on all Nacos nodes is eventually consistent.

Horizontal expansion

One of the shortcomings of ZK is that it cannot be scaled horizontally. This is a major problem of CP. With the development of the company and the larger scale, it is difficult for you to support the current business. There is no leader role in Distro, and each node can handle reading and writing. In this way, we can arbitrarily horizontally expand Nacos nodes to meet our needs.

In Distro, not every node can handle all read requests, but not every node can handle write requests. Each node will judge whether it should be handled by itself according to the hash value of the key. The write request accesses the domain name, which will hit each node randomly. How does Nacos make these write requests hit the corresponding machine? The answer is in DistroFilter:

This ServletFilter will do some filtering on each request. If it finds that the request is not its own, it will forward the request to the corresponding server for processing, and then return it to the user after receiving the result.

Here Nacos can actually do an optimization. We can find that this action is synchronous when forwarding. We can use asynchronous sending here, and enable the asynchronous of serlvet. This forwarding node can be similar to the gateway without synchronous waiting, and Nacos can be added. Throughput of the cluster

Compared with other CP protocols, these advantages of Distro are very large in the registry, and the implementation of the entire protocol is much simpler than them, which is very easy to understand. If you are involved in the registry protocol in the future, you can refer to this idea. .

Raft

In fact, there is also a strong consistency protocol in Nacos. Raft is used. Raft is used in two places:

In the registry, there is some data in Nacos that needs to be persistently stored. We will use Raft to store data consistently and synchronously, such as Service, some data in the namespace. Nacos thinks that the instance is a relatively fast-changing, temporary data , Raft is not required for a highly consistent protocol, but Service and namespace are data that changes less and are suitable for persistent storage. Raft implementation of the registry. At present, I am using a set of Raft written by myself. After reading the details, there are still some differences from standard Raft. For example, the continuity of logs is not guaranteed in the Raft protocol of Nacos.
After 1.3.0, in order to gradually use the standard raft protocol to implement the sofa-jraft, Nacos was only used in the configuration center for the time being. Before the configuration center, only mysql storage could be used. After 1.3.0, Nacos borrowed the passage of Etcd. The Raft protocol transforms the stand-alone KV storage into a distributed KV storage design idea. Based on SOFA-JRaft and Apache Derby, a lightweight distributed relational database is built, while retaining the ability to use external data sources. Users can Choose one of the data storage solutions according to your actual business situation.

The specific details of the Raft protocol will not be elaborated here. If you are interested, you can take a look at the translated paper: https://www.infoq.cn/article/raft-paper

Node registration and subscription

Another important thing in the registry is how our nodes register and subscribe, how to perform heartbeat detection to prevent node downtime, and how subscribed nodes can receive updates in real time.

Node registration

naming.registerInstance("microservice-mmp-marketing", "11.11.11.11", 8888, "TEST1");

We only need to simply write the above line of code to complete the registration of our node. The above code is for ServiceName microservice-mmp-marketing. Under TEST1Clsuter, an instance with IP as 11.11.11.11and port as 8888is registered, and a heartbeat task will be added at the same time of registration.

As shown in the red part of the figure above, a delayed heartbeat task is added to the thread pool, which is executed in 5s by default.

In BeatTask, the heartbeat will be sent to Nacos-Server first. If Nacos-Server defines the heartbeat interval, the next execution time will be modified according to the returned result. If our service does not exist, it means that our machine is due to a certain For some reason, the heartbeat has not been synchronized, which has caused it to be removed. Here, the node needs to be registered again.

In the ClientBeatCheckTask of Nacos-Server, we will regularly scan the Service for instances that have not been synchronized for a long time according to the Service dimension. The default is 15s, as shown in the code in the red box below:

Node's subscription

The subscription of nodes has different implementations in different registration centers. The general routines are divided into two types: rotation training and push.

Push means that when the subscribed node is updated, it will actively push to the subscriber. Our ZK is the implementation method of push. The client and the server will establish a long TCP connection, the client will register a watcher, and then when there is a When the data is updated, the server will push it through a long connection. Through this mode of establishing a long connection, the resources of the server will be seriously consumed, so when there are many watchers and when updates are frequent, the performance of Zookeeper will be very low, or even hang.

Rotation training means that the nodes we subscribe to actively and regularly obtain the information of the server nodes, and then do a local comparison, and if there are changes, some updates will be made. There is also a watcher mechanism in Consul, but unlike ZK, it is implemented through Http long polling. The Consul server will immediately return whether the requested url contains the wait parameter, or hang first and wait for the specified If there is a change in the service within the wait time, it will return. The performance of using round-robin training may be higher but the real-time performance may not be very good.

In Nacos, these two ideas are combined to provide both rotation training and active push. Let's first look at the rotation training part, and then in the run method in the UpdateTask class:

Pay attention to the red box above, we will regularly rotate training according to the Service dimension, our ServiceInfo, and then update it.

Let's take a look at how Nacos implements the push function. Nacos will record our subscribers to our PushService. The following is the core push code in our PushService:

Step 1: Trigger our push through the ServiceChangeEvent event. It should be noted here that because our nodes are updated through distro, this event will also be triggered when our distroT is synchronized to other machines.
Step 2: Get the subscribers maintained on the local machine, because the subscribers are defined according to whether the service node has been queried, and the action of querying the service node will be randomly hit to different Nacos-Servers, so each of our nodes Some subscribers will be maintained, and there will be duplications between the maintained subscribers. Since the subsequent UDP transmission is performed, the cost of repeatedly maintaining subscribers is not very high.
Step3: Generate ackEntry, that is, the content we send and cache it. The cache here is mainly to prevent repeated compression.
Step4: Finally send udp

The push mode of Nacos will save a lot of resources for Zookeeper's long connection through tcp. Even a large number of node updates will not cause too many performance bottlenecks in Nacos. In Nacos, the client will return if it receives a udp message. An ACK, if Nacos-Server does not receive an ACK for a certain period of time, it will re-send it. When it exceeds a certain re-send time, it will not be re-sent. Although it is not guaranteed to be sent to the subscriber through udp, but Nacos also has regular rotation training as the bottom line, so there is no need to worry about the situation that the data will not be updated. Through these two methods, Nacos not only ensures real-time performance, but also ensures that data updates will not be missed.

Summarize

Although Nacos is a new open source project, its architecture and source code design are very sophisticated. For example, a lot of EventBus architecture is used in the internal source code, which decouples many processes, and its Distro protocol idea is also very worth learning.

There are some other details in the Nacos registry, such as Selector based on tags and so on. Those who are interested can come down and find out in the Nacos documentation.

If you think this article is helpful to you, your attention and forwarding are the greatest support for me, O(∩_∩)O:

What You Should Know About Nacos Registry