Dubbo Cloud Native Road

 [Dubbo Cloud Native Road] Series Beginning

Author: Liu Jun

The flower name tortoise, the Github account Chickenlj, the Apache Dubbo PMC, the core development of the project, witnessed the entire process of Dubbo's restart of open source and graduation from the Apache Foundation. He is currently working in the cloud native application platform team of Alibaba Cloud, participating in service framework and microservice related work. Currently, he is mainly promoting Dubbo 3.0 - Dubbo cloud native.

Throughout the history of open source in China, you really can't find a second open source project with the same controversy and discussion as Dubbo.

On the one hand, in 2011, its open source filled the gap of the RPC framework used in the production environment at that time, and it was widely adopted as soon as it was released; Graduate with top-level program status. Even if Ali strives to show its determination to invest in open source, how will Dubbo go on in the face of the popular rising star Spring Cloud and the newborn Service Mesh? In the cloud-native era, how can it continue its current glory?

This year marks the one-year anniversary of Dubbo's graduation from the Apache Foundation, as well as an important year for advancing Dubbo 3.0, a full embrace of cloud native.

Dubbo and Open Source China jointly plan a series of articles [Dubbo Cloud Native Road] , and review the development of the Apache Dubbo community with you. The series of articles mainly cover the three parts of Dubbo technical interpretation, community operation, and application case analysis, and will meet you every week after that.

Here, we also invite all Dubbo users and developers to contribute. If you are using Dubbo or contributing to Dubbo, please share your development and use experience with us, and high-quality articles will also be included in [Dubbo Cloud] The original road] series.

Submission address: [email protected]

[Dubbo Cloud Native Road] The first part of the series: 3.0 is fully rolled out, the first anniversary of ASF graduation

This article, as the beginning of the entire series, will review and look forward to the development of the Dubbo project as a whole, and briefly summarize the subsequent articles.

From 2019 to the present, in the year since Dubbo graduated, the Dubbo community and products have made great progress. At the same time, the development of Dubbo cloud native version - Dubbo 3.0 has also been fully rolled out.

community. There are two points that need to be highlighted: one is that the number of corporate users who have landed and contributed has further increased, and more than 200 medium and large-scale companies have actively contacted the community, such as Ctrip, ICBC, Guazi Used Car, Netlink Clearing, Zhongtong, etc.; another sub-community represented by Dubbo-go is flourishing.

Product technology evolution. Dubbo Java Edition has released 10 versions, with in-depth exploration in multi-language, protocol, performance, service governance model, etc. Dubbo go has released more than 8 versions, and on the basis of basically aligning the functions with the Java version, some directions have also been ahead of the Java version. It is worth mentioning that Alibaba is also actively promoting the internal implementation of the Dubbo community version, and has gradually replaced its internal HSF framework with Dubbo starting this year. On the one hand, it is beneficial for Alibaba to export its rich service governance experience on HSF to the community. On the other hand, the official implementation of Alibaba will directly accelerate the development of Dubbo cloud native.

Under the tide of cloud native, 3.0 has been officially listed as the core goal of Dubbo product construction this year, involving the next-generation RPC protocol, service governance model, cloud native infrastructure adaptation and other aspects. Among them, many aspects have been pre-explored in the current 2.7 version, such as the recently released HTTP/2-based protocol support, application-level service discovery, etc., and follow-up work will be carried out on this basis. The series of articles will also have a detailed analysis of the Dubbo 3.0 Roadmap and technical solutions.

Dubbo's 1st Anniversary Review

In July 2017, the Dubbo open source project was reactivated and donated to the Apache Foundation in 2018. In May 2019, Dubbo officially graduated from the Apache Foundation incubation and became an Apache top-level project. Next, the article introduces Dubbo's achievements in the past year from three aspects: community, sub-community and product.

The community releases 24 versions a year, with over 300 contributors

If the initial reactivation was a project maintenance investment led by Alibaba, then since Dubbo joined Apache, it has become a completely open foundation project led by the community and mainly contributed by the community.

Today, this trend is becoming more pronounced. Internet and traditional enterprise companies including Alibaba, Ctrip, Industrial and Commercial Bank of China, Guazi Used Cars, Netlink Clearing, ZTO, etc. have invested in the use of Dubbo and community code contributions. The Dubbo community is becoming very active and diverse.

In the past year, the Dubbo community project has released a total of 24 versions and developed 27 Committer/PMC people, of which 20% of the contributors are from Alibaba, and more than 80% are developers or enthusiasts from different organizations.

The Dubbo community has organized more than 10 offline meetup events, basically covering cities where domestic developers gather. Through offline or online live events, share speeches on more than 100 topics, in-depth explanation of the latest developments in the Dubbo community, functional module development and recent plans. Most of the keynote speeches are collected by the community, and Dubbo's in-depth enterprises share practical experience. Typical representatives include Ctrip, Industrial and Commercial Bank of China, Koala, and credit computing power.

Judging from Github statistics, the number of Dubbo Stars has reached a new milestone, which has exceeded 30,000, an increase of nearly 5 times compared with the restart of open source; the number of contributors has increased from the initial dozens to more than 300 now, and among them More than 60 people have been nominated as committers, both the number of contributors and the ratio of committers have been greatly improved; Dubbo Java has released 65.

The above is mainly a summary of the development of the Dubbo Java project community. The following will introduce the progress of Dubbo Java products.

Dubbo Java iteration, currently maintains 3 major versions

There are currently three major versions of Dubbo Java maintained by the community , namely 2.5.x, 2.6.x and 2.7.x.

• 2.7.x is the main development version of the community. In the past year, 8 versions (2.7.0 - 2.7.7) have been released. Each version has some interesting features or functional upgrades, covering from programming model, Enhancements to multiple aspects of service governance, performance, and protocols.
• The 2.6.x version is positioned as a bugfix version. In the past year, a total of 3 versions have been released, mainly focusing on fixing problems and security vulnerabilities, and not adding too many new features.
• The 2.5.x version has announced EOF since the beginning of 2019 and only made security fixes; in the second half of the year, maintenance has been completely stopped.

The following is a brief hierarchical module diagram to review the evolution of Dubbo's technical architecture over the past period of time, from the perspectives of programming model, service governance, transmission protocol, performance optimization, etc.:

Many of the above functions have been implemented by major manufacturers to solve specific business problems. We also look forward to more in-depth summaries of Dubbo's practical experience from these manufacturers in the future.

The fifth year of Dubbo-go's development, it is going hand in hand with Dubbo

In addition to Dubbo Java, there are many excellent sub-projects (sub-communities) developed around Dubbo, including Dubbo-spring-boot-project , Dubbo-go , etc. Here we will focus on the Dubbo-go sub-community.

The Dubbo-go project was first built in May 2016 due to rain, and released and open sourced in September of the same year. The following timeline chart clearly records the past and present of Dubbo-go .

Adhering to the natural mission of "bridge the gap between Java and Go", Dubbo -go has entered its fifth year and has also embarked on its own unique development path:

• The current v1.4.0 version has been aligned with the 2.6.x version, the upcoming version will be aligned with v2.7.x [target v2.7.5], and then the v1.6.0 version targeting Dubbo 3.x will be released;
• Independently maintain a complete set of implementations from the underlying hessian2 protocol library Dubbo-go-hessian2 , the network library getty to the upper -level Dubbo-based Dubbo-go ;
• Independent TCP + Protobuf and gRPC + JSON communication solutions have also been developed [will be included in version v1.5.0];
• Interconnected with Dubbo/ gRPC / Spring Boot ;
• By accessing Opentracing and Promethus, Dubbo-go has carried out its own unique exploration in the direction of microservices such as observability;
• HTTPS-based trusted RPC calls have been implemented ;
• Has implemented its own unique microservice solution that uses kubernetes as a registry ;

Dubbo-go has developed into the most powerful Dubbo multi-language version from the very beginning of Dubbo's Go language implementation. Its development is inseparable from the strong Dubbo-go community behind it . In addition to the above-mentioned characteristics of Dubbo-go, through cross-community cooperation, the following achievements have been achieved:

• Through cooperation with the MOSN community, it has been realized that Dubbo/Dubbo-go applications can access Dubbo Mesh based on MOSN at zero cost , and realize "dual-mode micro-services" where micro-services and cloud-native coexist ;
• Cooperate with the sentinel community to fully access sentinel's downgrade and current limiting solutions in Dubbo/Dubbo-go ;
• Cooperate with the Apollo community to implement remote configuration distribution in Dubbo-go ;
• Cooperate with the Nacos community to realize service discovery based on Nacos; 

The main goals of the Dubbo-go community in Q2 2020 are:

• Release v1.5.0 version fully aligned with Dubbo 2.7.x ;
• Release the v1.6.0 version of Dubbo 3.0;
• Continue your own exploration in cloud native;
• Continue to maintain cooperation and progress with brother communities and expand the scope of their own use;
• In terms of production practice, we will promote the implementation of Alibaba Group and more manufacturers.

The project (including sub-projects) has been produced and implemented in companies such as Ctrip , Tuya Smart and Ant Financial .

After Ali Group completes the integration of HSF and Dubbo this year, the project will also be tested in Ali Group's Double Eleven battlefield.

Cloud Native Dubbo - Dubbo 3.0

3.0 is the codename for the next-generation Dubbo architecture. A year ago, when Reactive Stream was first explored, the community had extensive discussions about Dubbo 3.0. This time, in the context of cloud native, 3.0 represents a more comprehensive Dubbo architecture upgrade, involving the next-generation RPC protocol, a new service governance model, and cloud-native infrastructure adaptation.

Alibaba is one of the main forces involved in the development and construction of Dubbo 3.0. This open source project that started in Alibaba is returning to Alibaba's internal implementation.

Since last year, Alibaba has been gradually promoting the replacement of its internal HSF framework with Dubbo. By integrating the two frameworks of Dubbo and HSF, and on this basis, a Dubbo version adapted to the cloud-native architecture has been developed. The return of Dubbo to Alibaba is a good practice to embrace the community, embrace cloud native, and embrace standardization. The implementation of Dubbo 3.0 within Alibaba is also a major benefit to the community. On the one hand, it will help Alibaba to export its rich service governance experience on HSF to the community. On the other hand, it will directly promote the rapid development of Dubbo's cloud-native architecture. evolution. In addition to Alibaba, Douyu, Industrial and Commercial Bank of China, iQiyi, Douyu and other manufacturers are also participating in the construction of the next-generation Dubbo 3.0.

The three important directions in Dubbo 3.0 are listed below, and the specific Roadmap will be explained separately in the next article:

• Next-generation RPC protocol. The new protocol will provide richer built-in semantics such as Stream and Flow Control, as well as better scalability and gateway friendliness;
• Service discovery mechanism based on application granularity. On the basis of taking into account the ease of use and functionality of Dubbo's interface, it solves the problem of adaptation with Kubernetes Native Service, and solves the performance bottleneck of address push in large-scale clusters;
• Solutions for cloud-native infrastructure. This involves the connection between the Dubbo service and the infrastructure life cycle, the adaptation of the Kubernetes Native Service, the service governance rules for infrastructure scheduling, and the solutions for adapting to the Service Mesh architecture, etc.;

The following is a brief overview of these three aspects.

Next Generation RPC Protocol

Focusing on the protocol itself, the next-generation protocols mainly focus on HTTP/2, Reactive Stream, Flow Control, etc.:

• Reactive Stream Reactive Stream introduces RPC, which brings richer communication semantics and API programming model support, such as Request-Stream, Bi-Stream, etc.
  
• In the cloud-native scenario of HTTP /2 microservices, the communication protocol based on HTTP/2 has better versatility and penetration
  
• The Flow Control protocol has a built-in flow control mechanism and supports a Request (n) flow control mechanism similar to Reqctive Stream
  

From the perspective of the business scenario to be solved, based on the new protocol Dubbo should support load balancing algorithms for intelligent decision-making at the framework level, be more friendly to Mesh and gateways, and more easily provide multi-language implementation and interoperability, etc.

• Mesh protocol is more friendly to penetrating mesh, distinguishing protocol header Metadata and RPC Payload, which facilitates cooperation with mesh, including flow control mechanism, application layer configuration negotiation, etc.
  
• Protocol versatility Taking into account both versatility and performance, the support protocol can run on various devices
  
• Multilingual support, such as providing more complete cross-language service definition and serialization support by supporting Protobuf
  

Application-level service governance

Interface orientation has always been a strength of the Dubbo framework. On the one hand, its ease of use shields the existence of remote calls for developers; on the other hand, the interface-oriented address discovery and service governance bring more powerful capabilities, making the entire Dubbo governance system very powerful and flexible.

Since interface orientation has so many benefits, why should we explore application-oriented service governance models?

It sounds contradictory. In fact, whether it is interface-oriented or application-oriented, we just look at Dubbo from different perspectives. The transformation of "interface-oriented -> application-oriented" we are talking about is mainly reflected in the level of service registration and discovery :

The new application-oriented model we are talking about is mainly for the second point, that is, the data organization of the registry is changed to an "application/instance-oriented" granularity. This solves two problems for us:

• Aligns with microservice models such as Kubernetes Service at the service discovery level
• The amount of data discovered by the service will decrease by an order of magnitude, from "number of interfaces * number of instances" to "number of applications * number of instances"

For details, please refer to the article " Dubbo takes an important step towards cloud native - application-level service discovery and analysis ", and this series of articles will have a more in-depth analysis of this part of the mechanism and implementation.

cloud native infrastructure

Cloud native brings a full range of changes to the underlying infrastructure, application development, deployment, and operation and maintenance:

infrastructure

• Changes in the infrastructure scheduling mechanism bring changes in operation and maintenance (life cycle) and service governance.
• The service discovery capability sinks, and Kubernetes abstracts Native Service Discovery.

Service Mesh - Cloud Native Microservice Solution

• Mesh provides solutions for cross-language, sdk upgrade, etc. Dubbo sdk needs to cooperate with Mesh to achieve convenient adaptation of functions, protocols, and service governance.
• Mesh has not yet been rolled out on a large scale, and it is more suitable for applications that pay more attention to traffic control. The performance advantages of traditional SDKs still exist, and the co-location migration scenario of the two may exist for a long time.

In terms of application scenarios, possible deployment environments for Dubbo include:

1. Instead of using Kubernetes Native Service, Kubernetes is only used as a container orchestration and scheduling facility, and continues to use Dubbo's self-built service registration and discovery mechanism.
2. Reusing Kubernetes Native Service, Dubbo no longer cares about service registration, and Dubbo Client is responsible for service discovery and traffic distribution.
3. When Dubbo sdk migrates to Mesh, on the one hand, it needs to adapt to the Mesh architecture and become an RPC programming and communication solution under the Mesh system; on the other hand, it needs to achieve long-term coexistence of Dubbo and Mesh architectures and open up service discovery and governance systems to each other.
4. Smooth migration support for hybrid deployments on Kubernetes and off-cloud, including the unification of service discovery and the integration of network communication solutions.

From the function division of Dubbo, it will focus on providing support for cloud native infrastructure from the following aspects:

Life cycle: Dubbo is bound to the Kubernetes scheduling mechanism to maintain the automatic alignment of the service life cycle with the life cycle of Pod containers and other life cycles

Governance rules: Service governance rules are optimized in terms of rule body and rule format. For example, the rule body is described in YAML, the direct dependence of filtering rules on IP is canceled, and the CRD resources specific to the rules are defined.

Service discovery: Support service discovery of K8S Native Service, including DNS, API-Server, and service discovery of xDS

Mesh architecture collaboration: build a next-generation HTTP/2-based communication protocol and support the standardized data distribution of xDS

The new generation of RPC protocol and application-level service discovery model will be the pre-foundation of this part.

Summary and Outlook

As the beginning of the series of articles, here we briefly summarize and review Dubbo's achievements in the past year, including the development of the Dubbo community and product iterations.

Next, we will see more experience sharing from deep Dubbo users, development stories of Dubbo-go sub-communities, etc.

More importantly, we also look forward to the next generation of cloud-native Dubbo - Dubbo 3.0, and the follow-up Dubbo 3.0 Roadmap, scheme design and progress analysis will also be released in this series. 

Stay tuned!

{{o.name}}

{{m.name}}