Article directory
foreword
Service discovery is an old topic. Service discovery was born when applications began to run and access out of a stand-alone machine. The current network architecture is that each host has an independent IP address, so service discovery basically obtains the IP address deployed by the service in some way. The DNS protocol is the earliest protocol to translate a network name into a network IP. In the initial architecture selection, DNS+LVS+Nginx can basically satisfy all RESTful service discovery. At this time, the service IP list is usually configured in nginx or LVS. Later, RPC services appeared, and the online and offline services became more frequent. People began to seek a registration center product that could support dynamic online and offline and push IP list changes.
The Internet software industry generally favors open source products because the code of open source products is transparent, you can participate in co-construction, and there is a community for communication and learning. Of course, more importantly, they are free. Individual developers or small and medium-sized companies often choose open source products as their first choice. Zookeeper is a classic service registry product (although its original positioning is not here), for a long time, it was the only choice that Chinese people thought of when they mentioned the RPC service registry, which is a big deal. The degree is related to the popularity of Dubbo in China. Both Consul and Eureka appeared in 2014. The design of Consul includes many functions needed for distributed service governance, which can support service registration, health check, configuration management, Service Mesh, etc. Eureka has also gained a large number of users by virtue of the popularity of the concept of microservices and the deep integration of SpringCloud ecology. Nacos, which was open-sourced last year, carried Alibaba's large-scale service production experience and tried to provide users with a new choice in the market of service registration and configuration management.
An advantage of open source products is that developers can read the source code, understand the product's functional design and architectural design, and at the same time test the performance through local deployment, followed by comparison articles of various products. However, the current comparison of registration centers often stays in the comparison of superficial functions, and there is no very in-depth discussion on architecture or performance.
Another phenomenon is that the service registry is often hidden behind the service framework as a silently supported product. An excellent service framework often supports multiple configuration centers, but the selection of the registration center is still strongly related to the service framework. A common situation is that a service framework will have a default service registration center. Although this saves users from the trouble of model selection, the limitations of a single registration center cause users to deploy multiple sets of completely different registration centers when using multiple service frameworks. The data collaboration between these registration centers is also a question.
This article deeply introduces the design principles of the Nacos registry from various angles, and tries to summarize and explain the main points that should be followed and considered in the product design of the service registry from our experience and research.
1. Data model
The core data of the registration center is the name of the service and its corresponding network address. When the service registers multiple instances, we need to filter unhealthy instances or distribute traffic according to some characteristics of the instances. Some attributes such as health status and weight are stored on it. With the expansion of the service scale, it is gradually necessary to set some permission rules at the entire service level and some switches that are effective for all instances, so some attributes will be set at the service level. Later, we found that a single service instance would need to be divided into multiple subsets. For example, if a service is deployed in multiple computer rooms, it may be necessary to configure different instances of each computer room. Another data level is set between the service and the instance.
Zookeeper does not design a data model for service discovery. Its data is organized in a more abstract tree-shaped KV, so data of any semantics can be stored theoretically. Eureka or Consul both achieve instance-level data expansion, which can satisfy most scenarios, but cannot satisfy large-scale and multi-environment service data storage. The data model extracted by Nacos after years of internal production experience is a three-layer model of service-cluster-instance. As mentioned above, this can basically satisfy the data storage and management of services in all scenarios.
Although the data model of Nacos is relatively complex, it does not force you to use all the data in it. In most scenarios, you can choose to ignore these data attributes. At this time, you can reduce the dimension to the same data model as Eureka and Consul .
Another thing to consider is the data isolation model. As a shared service component, it needs to be able to ensure data isolation and security when used by multiple users or business parties. This is a slightly larger business very common in the scene. On the other hand, the service registry often supports deployment on the cloud. At this time, the data model of the service registry is required to be able to adapt to the general model on the cloud. Zookeeper, Consul, and Eureka do not have a clear model for service isolation at the open source level. Nacos has considered from the very beginning how to enable users to isolate data in multiple dimensions and at the same time smoothly migrate to the corresponding service on Alibaba Cloud. commercial product.
Nacos provides a four-layer data logic isolation model. The user account may correspond to an enterprise or an independent individual. Generally, this data will not be transparently transmitted to the service registry. A user account can create multiple namespaces, and each namespace corresponds to a client instance. The physical cluster of the registry corresponding to this namespace can be routed according to the rules, so that the internal upgrade and migration of the registry can be controlled. Users are imperceptible, and at the same time, physical clusters with different service levels can be provided to users according to their level. Further down is a two-dimensional service identifier composed of service grouping and service name, which can satisfy service isolation at the interface level.
Another new feature introduced by Nacos 1.0.0 is: temporary instance and persistent instance. The key to the definitional distinction between ephemeral and persistent instances is the way health checks are performed. Temporary instances use client-side reporting mode, while persistent instances use server-side reverse detection mode. Temporary instances need to be able to automatically remove unhealthy instances without persistent storage instances, so such instances are suitable for Gossip-like protocols. The persistent instance on the right uses the health check method of server-side detection, because the client will not report the heartbeat, so it is naturally impossible to automatically remove the offline instance.
In large and medium-sized companies, both types of services are often available. Some basic components, such as databases and caches, often cannot report heartbeats. When registering this type of service, it needs to be registered as a persistent instance. For upper-level business services, such as microservices or Dubbo services, the Provider side of the service supports adding logic for reporting heartbeats. At this time, the dynamic service registration method can be used.
Nacos 2.0 continues to use persistent and non-persistent settings, but with some adjustments. Persistent and non-persistent attributes in Nacos 1.0 are stored and identified as metadata of an instance. As a result, both persistent instances and non-persistent instances can exist under the same service. However, in actual use, we found that this mode will bring great confusion and complexity to operation and maintenance personnel; at the same time, from the perspective of system architecture, a service has both persistent and non-persistent instances There is also a certain contradiction in the scene. This leads to the fact that this ability is not widely used. In order to simplify the service data model of Nacos, reduce the complexity of operation and maintenance personnel, and improve the ease of use of Nacos, in Nacos2.0, the persistent data is abstracted to the service level, and no longer allows a service to exist at the same time. Instance and non-persistent instance, the persistent attribute of the instance inherits from the persistent attribute of the service.
2. Data Consistency
Data consistency is an eternal topic in distributed systems, and the complexity of the Paxos protocol makes data consistency a common topic for programmers to brag about. However, from the perspective of the agreement, there has been no new members joining the consensus model for a long time. At present, it can basically be classified into two types: one is the single-point write consistency based on the leader-based non-peer deployment, and the other is the multi-write consistency of the peer-to-peer deployment. When we choose the service registration center, there is no protocol that can cover all scenarios. For example, when the registered service node does not send heartbeats to the registration center regularly, the strong consensus protocol seems to be the only choice, because it cannot pass The heartbeat is used to perform data compensation registration, and the first registration must ensure that the data will not be lost. And when the client sends heartbeats regularly to report the health status, the success rate of the first registration is not very critical (of course it is also critical, but relatively speaking, we tolerate a small amount of data writing failure), because the follow-up can still pass If the heartbeat compensates the data, the single-point bottleneck of the Paxos protocol will not be cost-effective. This is why Eureka does not use the Paxos protocol but uses a custom Renew mechanism.
These two data consistency protocols have their own usage scenarios, and different requirements for service registration will lead to the use of different protocols. It can be found here that the behavior exhibited by Zookeeper under the Dubbo system is actually more appropriate to use Eureka's Renew mechanism, because the Dubbo service registers with Zookeeper as a temporary node, and it needs to regularly send heartbeats to Zookeeper to renew the node and allow the service to go down When online, remove the corresponding node on Zookeeper. Although Zookeeper uses the ZAB protocol to ensure strong data consistency, it lacks the disaster recovery capability of the computer room and cannot adapt to some large-scale scenarios.
Nacos officially supports the coexistence of AP and CP consensus protocols in 1.0.0 because it needs to support the registration of multiple service types and has essential capabilities such as computer room disaster recovery and cluster expansion. 1.0.0 restructured the data reading and writing and synchronization logic, and separated the business-related CRUD from the underlying consistency synchronization logic. Then abstract the reading and writing of the business (mainly writing, because the reading will directly use the cache of the business layer) into the data type defined by Nacos, and call the consistency service for data synchronization. When deciding whether to use CP or AP consistency, use a proxy to forward through controllable rules.
The current consensus protocol implementations are CP consistency based on simplified Raft, and AP consistency based on the self-developed protocol Distro. Needless to say, the Raft protocol is written based on the Leader, and its CP is not strict, but it can guarantee half of the seen consistency, and the probability of data loss is small. The Distro protocol refers to the internal ConfigServer and the open source Eureka, and achieves basically the same without the use of third-party storage. The focus of Distro is to do some logic optimization and performance tuning.
3. Load balancing
Strictly speaking, load balancing is not a function of traditional registries. Generally speaking, the complete process of service discovery should be to obtain a list of service instances from the registry first, and then select some of the instances according to your own needs or access different service providers according to a certain traffic distribution mechanism , so the registration center itself generally does not define the access strategies of service consumers. Eureka, Zookeeper, including Consul, do not implement a configurable and scalable load balancing mechanism. Eureka's load balancing is done by ribbon (the latest SpringCloud official work alone), while Consul is done by Fabio. .
Inside the Alibaba Group, the opposite idea is used. Service consumers often don't care about the load balancing of the service providers they visit, they only care about accessing the services of service providers with the most efficient and correct access. Service providers, on the other hand, pay great attention to the deployment of their own traffic. The first reason is that Alibaba Group's internal service access traffic is huge, and a little carelessness will lead to abnormal traffic that will overwhelm the service provider's services. . Therefore, the service provider needs to be able to fully control the traffic allocation of the service, and can adjust it dynamically.
Server-side load balancing gives service providers stronger flow control rights, but it cannot meet the needs of different consumers who want to use different load balancing strategies. However, scenarios with different load balancing strategies do exist. The client-side load balancing provides this flexibility and provides more friendly support for user expansion. However, if the client load balancing strategy is not properly configured, it may lead to hotspots of service providers, or no service providers at all.
Regardless of whether load balancing is implemented on the service provider or on the service consumer, we see that the current load balancing has strategies based on weight, service provider load, response time, and tags. Among them, the client load balancing mechanism designed by Ribbon is mainly to choose the appropriate existing IRule, ServerListFilter and other interfaces to implement, or inherit these interfaces by yourself to realize your own filtering logic. Ribbon here uses two-step load balancing. The first step is to filter out service provider instances that will not be used, and the second step is to implement a load balancing strategy in the filtered service provider instances. Ribbon's built-in load balancing strategies are relatively powerful, and because it allows users to expand, this can be said to be a relatively good design.
The tag-based load balancing strategy can be very flexible. Both Kubernetes and Fabio have used tags to filter resources. Using tags can almost achieve service traffic allocation with any ratio and weight. However, the tag itself requires separate storage and read and write functions, whether it is placed in the registration center itself or connected to a third-party CMDB.
In Nacos version 0.7.0, in addition to providing load balancing based on health checks and weights, we also provide a new label load balancer based on third-party CMDB. For details, please refer to the CMDB function introduction article. Using a tag-based load balancer, it is currently possible to implement a traffic scheduling policy that gives priority to access to the same tag. In actual application scenarios, it can be used to achieve nearby access to services. This is very useful when your services are deployed in multiple regions. Using this label load balancer can support a lot of scenarios, which is not going to be introduced in detail in this article. Although the label expressions supported in Nacos are not rich at present, the syntax it supports will be gradually expanded. In addition, Nacos defines Selector as a unified abstraction for load balancing. Regarding Selector, due to space constraints, there will be a separate article for introduction.
What should an ideal load balancing implementation look like? Different people will have different answers. What Nacos tries to do is to combine server-side load balancing and client-side load balancing through a certain mechanism, provide user scalability, and give users full autonomy and easy usage. Load balancing is a big topic. When we pay attention to the load balancing strategy provided by the registration center, we need to pay attention to whether the registration center has the load balancing method I need and whether the usage method is complicated. If not, will it allow me to easily expand to achieve the load balancing strategy I need.
4. Health check
Both Zookeeper and Eureka implement a TTL mechanism, that is, if the client does not send a heartbeat to the registry within a certain period of time, the client will be removed. What Eureka does better is that it allows customizing the health check method for checking its own status when registering a service. This is a relatively good experience in the scenario where the service instance can maintain heartbeat reporting. In the two major systems of Dubbo and SpringCloud, it has also been cultivated as the default behavior of the user's mind. Nacos also supports this TTL mechanism, but it is somewhat different from the internal mechanism of ConfigServer in Alibaba. Nacos currently supports temporary instances to use the heartbeat reporting method to maintain activity. The default heartbeat sending cycle is 5 seconds. The Nacos server will set the instance as unhealthy after 15 seconds without receiving the heartbeat, and remove the temporary instance after 30 seconds. .
However, as mentioned above, some services cannot report heartbeats, but can provide a detection interface for external detection. Such services also exist widely, and in our experience, these services also have a strong need for service discovery and load balancing. The most common methods of server-side health checks are TCP port detection and HTTP interface return code detection. These two detection methods can support most health check scenarios because of the versatility of their protocols. In some other special scenarios, it may be necessary to implement a special interface to determine whether the service is available. For example, if the master and backup of the database are deployed, the master and backup of the database may be switched under certain circumstances. It is necessary to provide external access through the service name to ensure that the currently accessed database is the master database. The health check interface at this time may be a MYSQL command to check whether the database is the main database.
Client-side health checks and server-side health checks have some different concerns. The client health check mainly focuses on the way the client reports the heartbeat and the mechanism for the server to remove unhealthy clients. The server-side health check focuses on the detection method and sensitivity of the client and the mechanism of setting the client's health status. In terms of implementation complexity, server-side detection must be more complicated, because the server needs to execute the corresponding interface according to the health check method configured by the registration service, judge the corresponding return result, and make a good retry mechanism and thread Pool management. This is different from client detection, which only needs to wait for the heartbeat and then refresh the TTL. At the same time, the server-side health check cannot remove unhealthy instances, which means that as long as the registered service instances do not call the interface to actively log out, these service instances need to maintain the detection task of the health check, and the client can remove unhealthy instances at any time. Reduce the pressure on the server side.
Nacos supports both client-side health checks and server-side health checks, and the same service can switch health check modes. We believe that the diversity of this health check method is very important, so that various types of services can be supported, so that these services can use the load balancing capabilities of Nacos. The next step for Nacos is to implement the user extension mechanism of the health check method, whether it is server-side detection or client-side detection. This can support the user to pass in a request of business semantics, and then Nacos will execute it, so as to customize the health check.
5. Performance and capacity
Zookeeper is exquisitely designed, but this is obviously because there are a series of prerequisites. First of all, the writing logic of Zookeeper is to write KV, and there is no aggregation inside; secondly, Zookeeper abandons the basic functions of service discovery such as health check and friendly query interface. When it supports these functions, it obviously needs to add some logic, even Abandon the existing data structure; finally, the Paxos protocol itself limits the size of the Zookeeper cluster, and 3 or 5 nodes cannot handle large-scale service subscriptions and queries.
When evaluating the capacity, it is necessary not only to evaluate the existing service scale of the enterprise, but also to predict the expansion scale in the next 3 to 5 years. Alibaba's middleware internally supports the group's million-level service instances, and the challenges encountered in terms of capacity can be said to be no less than those of any Internet company. This capacity not only means the number of instances registered as a whole, but also includes the number of instances of a single service, the number of overall subscribers, and the QPS of the query. In the process of eliminating Zookeeper and Eureka internally in Nacos, capacity is a very important factor.
The capacity of Zookeeper, in terms of the number of storage nodes, can reach the million level. However, as mentioned above, this does not represent all the capacity. When a large number of instances go offline, Zookeeper’s performance is not stable. At the same time, the defect in the push mechanism will cause the resource usage of the client to increase, resulting in a sharp drop in performance.
When the scale of Eureka's service instances is around 5000, the problem of service unavailability has already occurred. Even in the process of stress testing, if the number of concurrent threads is too high, Eureka will crash. However, if the service scale is around 1000, almost all current registries can satisfy it. After all, we see Eureka as the registration center of SpringCloud, and we have not seen a wide range of capacity or performance problem reports in China.
In the open source version of Nacos, the support for service instance registration is about 1 million, and the number of services can reach more than 100,000. In the actual deployment environment, this number may vary due to differences in machine and network configurations and JVM parameters. Figure 9 shows a summary of the results of Nacos stress testing using version 1.0.0, which includes testing and statistics on capacity, concurrency, scalability, and latency.
6. Ease of use
Ease of use is also a piece of content that users pay more attention to. Although the product can be very advanced in terms of functional characteristics or performance, if the user's use cost is extremely high, it will also discourage users. Ease of use includes many aspects of work, such as whether the access of API and client is simple, whether the documentation is complete and easy to understand, whether the console interface is perfect, etc. For open source products, another factor is whether the community is active. When comparing the performance of Nacos, Eureka, and Zookeeper in terms of ease of use, users in the community are sincerely invited to provide all-round feedback, because after all, within the Alibaba Group, the usage scenarios for Eureka and Zookeeper are limited. According to the experience and research, Zookeeper is not easy to use. The client of Zookeeper is complicated to use. There is no model design for service discovery and corresponding API encapsulation, which needs to be handled by the relying party. The support for multiple languages is not very good, and there is no easy-to-use console for operation and maintenance management.
Compared with Zookeeper, Eureka and Nacos have improved a lot. These two products have clients for service registration and discovery, as well as starters based on the SpringCloud system, helping users to register and discover services at a very low cost without perception . At the same time, it also exposes a standard HTTP interface to support multi-language and cross-platform access. Both Eureka and Nacos provide official consoles to query service registration status. However, as Eureka 2.0 announced the cessation of development, Eureka should not have a relatively large investment in the optimization of user use, and Nacos is still under construction. In addition to the ease of use features currently supported, the follow-up will continue Enhance the capabilities of the console, increase the management and control of console login and permissions, monitor the system and the exposure of Metrics, continue to improve the use of documents through the official website and other channels, and develop multi-language SDKs.
From the perspective of community activity, due to the large number of existing users of Zookeeper and Eureka, many tutorials and troubleshooting can be found in the community. In this regard, the newly open source Nacos needs to continue to accumulate over time.
Seven, cluster scalability
集群扩展性和集群容量以及读写性能关系紧密。当使用⼀个比较小的集群规模就可以支撑远高于现有数量的服务注册及访问时，集群的扩展能力暂时就不会那么重要。从协议的层面上来说，Zookeeper 使用的 ZAB 协议，由于是单点写，在集群扩展性上不具备优势。Eureka 在协议上来说理论上可以扩展到很大规模，因为都是点对点的数据同步，但是从我们对 Eureka 的运维经验来看，Eureka 集群在扩容之后，性能上有很大问题。
Another aspect of cluster scalability is support for multi-region deployment and disaster recovery. When paying attention to the high availability and stability of clusters and the cross-regional delay on the network, it is required to deploy clusters in each region. Our existing solutions have multi-room disaster recovery, multi-active in different places, and multi-data centers.
The first is dual-computer room disaster recovery, which cannot be supported without modification based on the protocol written by the Leader, which means that Zookeeper cannot achieve dual-computer room disaster recovery without manual intervention. In the case of a single computer room disconnected from the network, it is not difficult to make the services available in the computer room. The difficulty is how to aggregate data after the network disconnection is restored. Zookeeper’s single-point write mode will have data reconciliation problems after the network disconnection is restored. Eureka's deployment mode naturally supports multi-computer room disaster recovery, because Eureka uses a registration mode of purely temporary instances: no persistence, and all data can be compensated through client heartbeat reporting. As mentioned above, both temporary instances and persistent instances have their application scenarios. In order to be compatible with these two scenarios, Nacos supports two modes of deployment. One is the deployment of the same AP protocol as Eureka. This mode only It supports temporary instances, can perfectly replace the current Zookeeper and Eureka, and supports computer room disaster recovery. The other is the CP mode that supports persistent instances. In this case, dual-room disaster recovery is not supported.
When it comes to remote multi-active, it is a coincidence that the remote multi-active of many business components is realized by the service registry and configuration center, which includes traffic scheduling and modification of cluster access rules. Disaster recovery in computer rooms is a part of multi-active in different places, but to enable business to dynamically adjust the cluster nodes accessed when accessing the service registry, this requires third-party components for routing. Multi-active in different places is often an overall plan that includes all product lines, and it is difficult to say whether a single product supports multi-active in different places.
In fact, multi-data centers can be regarded as a part of multi-living in different places. From the perspective of a single product, Zookeeper and Eureka did not give an official multi-data center solution. Based on Alibaba's internal experience, Nacos provides a solution that uses the Nacos-Sync component to synchronize data between data centers, which means that each data center's Nacos cluster will have full data from multiple data centers. Nacos-Sync is an important part of the Nacos ecological components. It will not only undertake data synchronization between Nacos clusters and Nacos clusters, but also undertake data synchronization between Nacos clusters and Eureka, Zookeeper, Kubernetes, and Consul.
8. User scalability
In the design of the framework, scalability is an important design principle. Frameworks such as Spring, Dubbo, and Ribbon have made relatively good designs in terms of user scalability. The extensibility of these frameworks often uses technologies such as interface-oriented and dynamic class loading to run user-extended interfaces and implement user-defined logic. In the design of Server, user extension is more prudent. Because the introduction of user extension code may affect the availability of the original Server service, and if there is a problem, it will be more difficult to troubleshoot. Well-designed SPIs are possible, but the resulting stability and operational risks need to be carefully considered. In open source software, user extensions are often realized by directly contributing code. Good extensions will be continuously updated and maintained by many people. This is also a relatively good development model. Zookeeper and Eureka currently do not support user extensions on the server side. A service discovery product that supports user extensions is CoreDNS. The overall architecture of CoreDNS is connected in series through plug-ins. By placing the plug-in codes under the CoreDNS project in an agreed manner, and rebuilding, the plug-ins can be added to the coreDNS overall function link.
So is such scalability necessary? To give an example mentioned above, if you want to add a new health check method, connect to the database and execute a MySQL command, the usual way is to add a MySQL-type health check method in the code, build, test, and finally release. But what if the user is allowed to upload a jar package and put it somewhere in the server deployment directory, and the server will automatically scan and recognize this new health check method? This is not only cooler, but also makes the entire extension process decoupled from the Server code, making it very simple. So for some functions of the system, if they can be opened to users to expand at runtime through careful design, why not do it? After all, adding extended support will not cause any loss of original functions.
All products should try their best to support user runtime extensions, which requires that the server-side SPI mechanism be designed to be sufficiently robust and fault-tolerant. In this regard, Nacos has opened up extended support for third-party CMDBs, and will soon open user extensions for core functions such as health checks and load balancing. The purpose is to be able to support various needs of users in a decoupled way.
end
This article is not an article introducing the functions of Nacos, so some features of Nacos are not covered in the article. These features are actually important aspects that distinguish Nacos from other registries, including the DNS protocol supported by Nacos, and custom labeling. and other capabilities. Readers who are slightly familiar with Nacos may notice that the overall architecture of Nacos is somewhat similar to that of Consul, but in fact, Nacos and Consul have basically no similarities except that they deploy configuration management and service discovery together. We'll cover the differences with Consul in another article. The architecture and functions of Nacos are derived from ten years of operating and evolution experience within Alibaba, so the comparison between the two will definitely let everyone know that their positioning and evolution direction are completely different.
Nacos 2.0 has released the GA version, and will continue to output new functions in the way of co-construction with the community in the future, and continue to cultivate in the two fields of service discovery and configuration management. We look forward to working with you to build the best service discovery and configuration management platform.