Dubbo 3.0 forward-looking docking of Kubernetes native services

Kubernetes is currently the most popular container service platform in the world. In a Kubernetes cluster, the deployment method of Dubbo applications often requires the use of a third-party registry to realize service discovery. The combination of Dubbo and Kubernetes' scheduling system can greatly reduce the operation and maintenance costs of the two platforms that originally needed to be managed, and Dubbo's adaptation of Kubernetes native services can also make the framework itself more integrated into the cloud native system. The new application-level service discovery model based on Dubbo 3.0 makes it easier to align with the service model of Kubernetes.

Kubernetes Native Service

In Kubernetes, Pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Usually a Pod consists of one or more containers, and the application is deployed in the container.

For a group of Pods with the same function, Kubernetes defines the abstraction of such a group of Pod strategies through the concept of Service, which is Kubernetes Service. These Pods marked by Kubernetes Service are generally determined by Label Selector.

In Kubernetes Service, service nodes are called Endpoints. These Endpoints correspond to the application units that provide services, and usually correspond to Pods one-to-one.

Therefore, we can use Service to schedule the applications in microservices, and the calls between microservices realize service discovery through a series of Service mechanisms, and then integrate microservices into the Kubernetes Service system.

Dubbo application-level service discovery

In the Dubbo architecture, in order to better conform to the programming habits of Java developers, the bottom layer of Dubbo uses interface granularity as the registered object. But this model is very different from the current mainstream Spring Cloud registration model and Kubernetes Service registration model.

At present, the registration model outside the non-Dubbo system is mainly based on the service granularity as the registration object. In order to break through the registration discovery barriers between Dubbo and other systems, Dubbo has introduced a service introspection architecture after version 2.7.5, mainly through metadata services Realize the transition from service granularity to interface granularity. From version 2.7.5 to version 3.0, the service introspection model has been optimized in many aspects and verified in a production environment.

1. Metadata Service

The metadata service mainly stores the mapping relationship between the service (Instance) and the interface (Interface). By abstracting the original interface information written in the registry into metadata for storage, on the one hand, the amount of data stored in the registry can be greatly reduced. , Reduce the network communication pressure of the cluster when the service is updated. On the other hand, it achieves the goal of storing only application granular information at the registry level, aligning with other registration models.

In the service introspection model, the service provider not only writes the current instance information to the registry, but also needs to write the exposed service URL information to the metadata service (local or remote); and for service consumers, After the registration center obtains the instance information, it also needs (through RPC request built-in or centralized configuration center to obtain) the metadata service to obtain the service URL information of the service provider, etc. to generate the interface information under the interface granularity system.

2. Revision Information

Revision information is a data caching mechanism introduced by metadata services. For the same set of applications, the exposed interfaces are actually the same in many cases. There will be a lot of redundancy when mapping services (Instance) and interfaces (Interface). Therefore, you can use a similar MD5 calculation method to add the version number to the instance itself. If the version numbers of multiple instances are the same, it can be considered that their metadata information is also the same, so you only need to randomly select one To obtain metadata information, the traffic can be compressed from the need to communicate with a group of instances to only one instance.

As shown in the figure above, the working mechanism of the service consumer registry can be summarized as:

The service consumer obtains a list of service instances from the registry.
The registry returns service instance information to the service consumer, and the instance list includes the Revision parameter written by the service provider to the registry.
Service consumers are grouped according to the Revision parameters of the obtained instance information, and one is randomly selected from each group of instances to obtain the metadata service.
The service consumer initiates a call through RPC or obtains the metadata information of the specified instance through the configuration center.
Service consumers construct service information with interface granularity according to the acquired metadata information.

For more information on application-level service discovery, please refer to: " Dubbo takes an important step towards cloud native-Application-level service discovery analysis "

Docking realization method

There are two ways to connect with Kubernetes this time. One is to obtain information in the form of Kubernetes API Client, and the other is to obtain information in the form of DNS Client.

1. Kubernetes API Client

The core of the Kubernetes control plane is the API Server, which provides HTTP APIs for users, different parts of the cluster, and external components of the cluster to communicate with each other. For Dubbo, communication with the Kubernetes control plane can be achieved by using the Kubernetes API Client.

1) Provider side details

According to the Dubbo application-level service discovery model mentioned above, the Provider side needs to write Revision information to the registry when the application starts and the interface is updated. Therefore, the general logic is shown in the figure above.

The Label tag is used as a Selector to match the Service. Annotation mainly stores Revision and other information. The Revision information needs to be written by the Dubbo application to the Kubernetes API Server. This also corresponds to the service registration process.

In the current version of the implementation, the creation of Kubernetes Service is implemented by the operation and maintenance side, that is, the Label Selector is managed by the operation and maintenance side, and it has been configured before the Dubbo application is started. The name of the Service is also It is the Services field in the corresponding interface annotation (for those who do not rely on any third-party configuration center, you need to manually configure this field at the interface level).

2) Consumer side details

The logic on the Consumer side is roughly the same as the design of the application-level service discovery model. After the service information is obtained through the API, the ServiceInstance information is supplemented by obtaining the Annotation information of the corresponding Pod, and the subsequent logic is consistent with service introspection.

3) Advantages and disadvantages

It should be pointed out that allowing the application itself to directly interact with the API of the Kubernetes management platform itself has certain security risks. If the configuration is incorrect, there is a certain possibility that the entire Kubernetes cluster will be brought down.
When a large number of applications are updated, it will put a certain pressure on the Kubernetes API Server.
This solution directly connects Dubbo's service discovery process to the management of the Kubernetes cluster, which can further simplify the management complexity in the Kubernetes environment.

2. DNS Client

Kubernetes DNS is a mechanism provided by Kubernetes to obtain Kubernetes Service information through DNS queries. Service node information can be obtained through ordinary DNS requests.

1) Fully decentralized metadata service

Due to the limitation of the DNS protocol itself, there is currently no unified and simpler way to add more information to the DNS for writing the Revision information. For the implementation of DNS, we modified the metadata service so that it no longer relied on writing Revision information to the registry, so that we only need to know the IP of the Dubbo application to achieve normal service discovery.

The modified metadata service can be divided into two major functions, one is the ability to obtain URL information based on the revision, and the other is the ability to obtain the corresponding revision. Dubbo service consumers will actively connect with the metadata service of each instance after obtaining the instance IP through the registration center. After obtaining the revision information, randomly select an instance with the same revision information to obtain the complete metadata information.
Since the Revision information is obtained in a point-to-point manner, when this information is updated, it must be notified to the consumer in time for the update. In the current version of the implementation, we rely on the parameter callback mechanism to implement the server's initiative to push to the consumer. In the future, it will realize streaming push based on the new Triple protocol of 3.0.

2) Provider side details

Similar to the implementation of the Kubernetes API Client, the process of building services needs to be performed by the operation and maintenance side. When the service provider starts, the metadata information will be cached locally and the metadata service interface will be exposed externally.

It is important to note here that in order to decouple the service discovery process from the business service itself, the metadata service interface and the corresponding port of the business interface can be inconsistent. When using the DNS implementation scheme, the metadata service ports of all applications in the entire cluster are required Unified, configured through the metadataServicePort parameter. In this way, it can also adapt to those DNS that do not support port acquisition through SRV.

3) Consumer side details

For the DNS registry, the process of obtaining an instance is mainly obtained by initiating an A (or AAAA) query request to DNS, and Kubernetes DNS also provides SRV record requests to obtain service information.

Combined with the fully decentralized metadata service mechanism mentioned in the previous article, Consumer will actively connect to the metadata service of each instance it obtains, obtain the corresponding Revision information, and submit a callback function to the provider in the form of parameter callback. To update local information.

After obtaining the Revision information, for instances with the same Revision information, Dubbo will randomly select one of them to obtain the complete metadata information, thus completing the whole process of service discovery.

4) Advantages and disadvantages

Compared with the previous one, this solution avoids the direct interaction of Kubernetes API and avoids the security problem of interaction.
Since DNS does not have a notification mechanism like API Watch, it can only use polling to determine service changes. There is still a certain amount of DNS network query pressure in the cluster when there is no application change.
When this solution was designed, the goal is to adapt to any DNS that can obtain the IP of the Dubbo application itself through A (or AAAA) query, and it is not a strong dependency on Kubernetes DNS.

to sum up

This time Dubbo docking with Kubernetes native services is an attempt by Dubbo to develop toward cloud-native development. In the future, we will implement interaction with the Service Mesh control plane based on the xDS protocol, and related functions are under intense planning. At the same time, in the future release of Dubbo 3.0, the Java community and the Dubbo-go community will be released simultaneously, and the functions of Kubernetes will also be launched this time.

About the Author

Jiang Heqing , Github account AlbumJ, Apache Dubbo Committer. As an undergraduate student, he is currently mainly involved in the docking of the cloud native Kubernetes and Service Mesh modules in the Dubbo community.