Author: Yanlin
This article is compiled from the online live broadcast of Yan Lin, senior technical expert of Alibaba Cloud Intelligence, to share "Best Practices of Cloud Native Microservices". Video playback address: https://yqh.aliyun.com/live/detail/28454
With the evolution of cloud native, microservices are widely used as the mainstream application architecture, and the difficulty of its implementation has gradually extended from how to build it to how to use it well. Today, I will share with you my 10 years of practical experience in the field of microservices, and how to make microservices solid in a more efficient manner.
Alibaba Microservice Development History
Microservice 1.0 (1w instance/microservice split/same city disaster recovery)
In 2008, as the scale of Alibaba's business continued to increase, the single fat application + hard load architecture gradually exposed performance bottlenecks; as the number of R&D personnel gradually increased, the coordination efficiency also gradually decreased, unable to meet the increasingly complex business challenges, so there was an urgent need for technology upgrades solve these problems.
At that time, the SOA architecture was very popular, and it became the main direction of our technology evolution. At that time, there were two solutions, one was the server-based solution, which was less intrusive and convenient for centralized control, but this centralized solution would bring The cost is high, the stability risk is high, and the scalability is poor; one is the Client-based solution, which is decentralized, highly scalable, and low-cost, but it will bring some intrusiveness and be difficult to manage; of course, many people will Why not just use DNS directly? The main reason is that DNS cannot meet the real-time performance of IDC's internal service discovery, and the service list update cannot be notified in time that there is a business, which will lead to loss of business traffic.
After evaluating the pros and cons of the two solutions, we adopted the Server Based solution in the gateway scenario that requires centralized management of security and simple routing, and evolved the Alibaba Tengine gateway technology system based on Nginx to solve security, high availability, and simple routing from the entrance. Capability; The Client Base model was adopted within IDC, and the HSF/Dubbo+Nacos technology system was incubated to support the splitting of business microservices.
With the implementation of the first-generation microservice architecture, the introduction of the registry brings stability risks, and the suspension of the registry will cause all the call links to be interrupted; when the business is released centrally, the registry will be under great pressure.
For usability problems, we provide the ability to push the air, even if the registration center hangs, it will not affect the normal operation of the business; in order to provide better performance, we provide a fully asynchronous architecture; in order to support the same city disaster recovery, we provide the AP consistency protocol, the specific protocol You can refer to the "Nacos Architecture and Principles" e-book.
With the implementation of Alibaba's microservices 1.0 architecture, it helps businesses complete microservices split, solves the problems of scalability and collaboration efficiency, and supports Alibaba's intra-city disaster recovery capabilities. For small partners who are working on microservices, you may ask how Alibaba does the evolution of the microservice architecture:
The separation of front-end and back-end is the first step, because the front-end has many changes and rapid changes, while the back-end has relatively small changes and slow evolution. Therefore, it is necessary to decouple development, so that the front-end can adapt to market changes more quickly, so as to maintain an advantage in the competition;
The stateless transformation of the backend is the second step. The memory state is externalized to Redis, and the persistent state is externalized to Mysql, so that the business can be divided at will;
The third step is modular splitting, which is the most challenging for architects, because splitting is based on business attributes and splitting based on application complexity. This is a relatively dynamic process. It is recommended to split modules. The last 2-3 people are responsible for a module. If it is disassembled too finely, it will lead to relatively high operation and maintenance costs. If it is disassembled too thickly, it will bring about R&D coordination problems. eventually reached a relatively steady state. It is worth mentioning here that one of the advantages of HSF/Dubbo, because the SOA architecture was adopted in the early stage, an interface is a service, which is actually very convenient for splitting and merging services. Of course, it also brings a problem. The performance pressure comparison of the registry Large, it's a matter of architectural choice and balance.
Microservice 2.0 (10w instance/business center/remote live)
The microservice 1.0 architecture has helped Alibaba greatly alleviate performance and efficiency problems. However, due to the success of Alibaba's Double Eleven, we are facing a flood of technical challenges. We must strike a balance between user experience, resource cost, and high availability. Our biggest challenge at this stage is scalability and stability. The scalability is to support the expansion of 10w+ instances of the business, but the resources in a single location are limited, and the double eleven merchants are investing more and more funds, so that we can't be out on the double eleven day. Serious problems, otherwise the losses will be very large, so very high requirements are placed on business stability.
Therefore, Alibaba's evolution to Microservice 2.0 supports a high-availability system with multiple activities in different places, allowing Alibaba's business to expand horizontally at the IDC level. New computer rooms and new technology systems can be verified in units, which also accelerates the evolution of Alibaba's technology system. speed.
During this period, the pressure of horizontal notification between Nacos Servers is huge, and the service release window is easy to fill up the network card. Frequent pushes will consume a lot of memory and CPU of the service, which will affect the stability of the service.
In response to the above problems, we made aggregate push between Nacos Servers, merged and aggregated the changes in a certain time window, and performed compressed push during the push process, thus solving the above problems.
While microservices solve scalability and high availability, there are more business systems, duplication of construction, more and more business islands, and lower and lower collaboration efficiency. Therefore, Alibaba Business has launched the business middle-office capability at this time, and will The flat microservice abstraction layering abstracts basic services into middle-office services to solve the above problems. After business layering, it supports the rapid growth of Alibaba's business and accelerates the unification of technical architecture.
Microservice 3.0 (100w instance/business domain split/cloud native)
The microservice 2.0 architecture supports the technological miracle of Alibaba's Double Eleven. Alibaba has also started to expand its business and build a more complete Internet map. At this stage, Ali has acquired a lot of companies, and the technical system is not unified how to form a joint force; after going from online to offline, the offline system has higher requirements for system stability; cloud computing development, how to make good use of cloud elasticity Double 11, at this stage, we have also launched cloud products with microservices, hoping to support Alibaba's Double 11 through cloud products.
Business domain segmentation is relatively easy, and how to better interconnect after the segmentation is a key, so we have launched two products, Nacos-sync and cloud native gateway. Nacos-sync is suitable for scenarios with large business traffic and consistent protocols. Cloud native gateways are suitable for scenarios such as network failure, different protocols, and cross-regions.
Even if the business domain is split from the top level, the evolution of the largest e-commerce cluster to millions of instances puts more and more pressure on the registry. We continue to lengthen the aggregation window time, and the slow push will lead to business release time. Longer and faster pushes will consume a lot of business, so you are caught in a dilemma.
At this stage, we decompose the problem. First, we make a segmentation according to the size of the service list. If the service list is large, it can be pushed slowly, and the problem is not big. If the service list is small, it needs to be pushed in time. Therefore, we have optimized the aggregation push logic, according to the service list. The list size is graded and pushed. Another optimization idea is to change only a few list changes, so we provide the incremental push capability, which greatly reduces the amount of service change push data.
Through the evolution of the microservice 3.0 architecture, the problems of cross-domain interoperability and smooth cloud migration are well solved. New services can be migrated to the cloud first, or some services can be migrated to the cloud, and the cloud-to-cloud intercommunication can be done through gateways. At the same time, it supports millions of Instance microservice architecture evolution.
I hope to provide you with some ideas and inspiration for the evolution of the microservice architecture through my sharing of the development process of Alibaba's microservices.
Cloud-native microservice trends
With the evolution of cloud native technology, containers take the concept of immutable infrastructure to solve the problems of operation and maintenance standards and resource utilization; microservices take the concept of variable runtime to solve the problem of R&D efficiency and improve the overall scalability and high availability of the system. People often ask me, why do we need a microservice registry when we have a service discovery mechanism for containers? First of all, the architecture is layered. When I was young, I really couldn’t see any obvious difference, but when I was older, problems would be discovered. For example, the largest microservice cluster in Alibaba Center, the bottom layer is multiple Kubernetes clusters, to prevent a Kubernetes problem from affecting the overall situation. The underlying Kubernetes can also be scaled horizontally. If you rely on the service discovery mechanism of Kubernetes, cross-Kubernetes service discovery becomes the first problem. Of course, the bottom layer is a Kubernetes, and it can also be multiple microservice environments. Microservices can be segmented according to business domains. The two layers can be decoupled and freely combined with the environment. In addition, Alibaba's microservice system has accumulated a complete system of push-air protection and service governance, while Kubernetes' CoreDNS forces service discovery to the business invocation link, and performs domain name resolution for each invocation. Therefore, when CoreDNS is suspended, all services are interrupted. As for Alibaba's overall evolution from millions of instances to tens of millions of instances, this part is also the content of Alibaba Microservices 4.0. This part has limited reference to most companies, so it will not be expanded.
Microservices Best Practices
After more than 10 years of development, Alibaba's microservice system has been widely used through open source, and has supported thousands of enterprises in digital upgrading through Alibaba Cloud. Take this opportunity to share our best practice summary with you, hoping to help you make good use of microservices.
Introduction to Alibaba Microservice System
Through MSE + ACK, the first step of cloud-native technology upgrade can be completed, releasing cloud elasticity dividends, releasing R&D efficiency dividends, and making better use of the microservice system through observability and high availability.
Microservices Best Practices
The microservice split is completed through the registration & configuration center; the security and high availability problems are solved from the entrance through the unified entrance through the gateway; finally, the problem of user microservices is improved through service governance.
Gateway Best Practices
As a next-generation gateway, the cloud native gateway provides a one-stop gateway solution with high integration, high availability, high performance and security.
• Unified access: The three-in-one traffic gateway, microservice gateway, and WAF greatly reduce resources and operation and maintenance costs. It should be emphasized that the solution of integrating WAF with cloud native gateway has very good performance advantages, and WAF is issued as the control plane. The protection rules are sent to the cloud native gateway, and the traffic is directly routed to the back-end machine after cleaning the cloud native gateway. The RT is short and the operation and maintenance cost is low.
• Unified entrance security defense line: automatically update certificates to prevent expiration, support JWT/OAuth2/OIDC/IDaaS authentication mechanisms, and support black and white list mechanisms.
• Unify east-west, north-south traffic: solve the problem of cross-domain intercommunication, including cross-network domain, cross-service domain, cross-region, and cross-security domain.
• Unified service discovery mechanism: supports various service discovery methods of Nacos/Kubernetes/DNS/fixed IP.
• Unified observation platform: Do a full-link diagnosis of tracing buried points from the entrance, enrich the business overview and alarm templates, and greatly reduce the cost of gateway operation and maintenance.
• Unified service governance: Provide high availability capabilities such as current limiting, downgrade, and fusing from the entrance, and provide a full-link grayscale solution to control change risks. Unified performance optimization: The performance of hardware acceleration is improved by 80%, the performance of Ingress scenarios is 90% higher than that of Nginx, and the performance of parameter tuning + module optimization is improved by 40%.
The cloud native gateway supports WASM to extend the gateway customization function, and provides rich plug-in capabilities through the plug-in market.
Service Governance Best Practices
Provide zero business intrusion, development, testing, operation and maintenance full coverage service governance capabilities, improve system high availability. For example, even if the registry is pushed in milliseconds during the release stage, there will be a delay, which will cause traffic loss during this period. Therefore, we provide lossless online and offline capabilities to solve this pain point. This month, we will open source our service governance capabilities through OpenSergo, and welcome all friends to participate in the joint construction!
Routine Environmental Isolation Best Practices
Sharing a set of environments, joint debugging and development affect each other, and the cost of joint debugging machines for all environments is too high. This is a contradiction. We use the full-link marking capability to isolate traffic, so that everyone can isolate multiple logical connections in one environment. Adjust the environment and solve this problem ingeniously.
Configuration Management Best Practices
As the application scale becomes larger, the cost of modifying the configuration and operation and maintenance of each machine is too high. Therefore, the configuration center needs to maintain the application configuration uniformly, make static services dynamic, dynamically modify the service runtime behavior, and improve the application runtime flexibility.
Service Mesh Best Practices
Small partners who have demands for multi-language development and are interested in service gateways can quickly build service grid solutions through MSE+ASM, complete service interoperability, and quickly experience new technologies.
High Availability Best Practices for Microservices
As business complexity increases and business peaks are unpredictable, it is very important to use high-availability tools in the face of failed designs and microservices. Sentinel can be used to complete the protection of current limiting, downgrade, and fuse, and PTS can be used to complete pressure testing. Complete destructive testing through chaos engineering, and improve the high availability of the system as a whole.
Practice of smooth migration of registry
At present, dual registration is recommended for large-scale scenarios, such as 1w or more instances, which has a longer release cycle and higher stability. If there are less than 1w instances, the registration center can be smoothed through Nacos-sync synchronization, which is more versatile.
Gateway Balance Migration Practice
Due to the three-in-one and performance advantages of the previous cloud native gateway, you can switch to the cloud native gateway through the ingress DNS grayscale.
Microservice benchmark customers
There are two types of typical customers when users go to the cloud, one is traditional monolithic fat application customers, and the other is users who have adopted microservices and need to make good use of microservices. We will share them through two benchmark customers.
Skechers Microservices + Business Middle Office Practice
It is very urgent for Skechers to find us for digital upgrade in 2021. It takes about 3 months before Double Eleven to complete the digital upgrade. Using MSE micro-service + middle platform solution, Skechers completed the east-west, north-south traffic with the help of cloud native gateway. With the help of the north-south cloud native gateway to complete security authentication and ingress current limiting, and to protect traffic from the ingress; with the help of the east-west gateway to complete the interworking of multiple business domains, the intercommunication between new and old systems is completed in about 1 month. The whole system was built, and the whole system stress test and high availability verification were completed in about a month, and the big promotion business was very successful in the end, helping Skechers Double Eleven's revenue scale of 1.2 billion.
Best practice of full-link grayscale for incoming call microservices
Technical challenges of incoming calls
Caller Technology has rich business scenarios and many systems. It has completed containerization and microservice transformation in its technical architecture. The microservice framework uses Spring Cloud and Dubbo. With the rapid development in recent years, the power bank equipment nodes and business volume are increasing rapidly, and the stability of the system faces several challenges:
1. How to avoid the loss of business traffic during the release of system services; 2. The system lacks simple and effective grayscale capabilities, and each system release has certain stability risks. MSE Microservice Governance provides out-of-the-box and non-intrusive online publishing stability solutions and full-link grayscale solutions, helping Caller Technology eliminate publishing risks and improve online stability.
Best Practices of Grayscale for All Links of Incoming Calls
1. Caller Technology chooses MSE Microservice Governance Professional Edition to achieve non-intrusive microservice governance capabilities, seamlessly supporting all Spring Cloud and Dubbo versions on the market in the past 5 years, without changing a line of code or changing the existing business architecture Just use it, no bindings.
2. The MSE Microservice Governance Professional Edition provides a full-link grayscale solution to help Caller Technology quickly implement the three-dimensional capabilities of grayscale, observable, and rollback safe production, to meet the needs of rapid iteration and careful verification under the condition of rapid business development demands;
3. The lossless online and offline capability of MSE microservice governance protects the entire process of system services. Through a series of capabilities such as service preheating, lossless offline, alignment with the Kubernetes microservice life cycle, and delayed release, it ensures that services are cold-started. Or in the process of destruction, business continuity is not damaged.
4. The outlier instance removal capability of MSE microservice governance enables service consumers to automatically detect the availability of the provider instances they call and dynamically adjust the weights in real time to ensure the success rate of service calls and improve business stability. sex and quality of service.
Alibaba Cloud Microservice Ecology and Planning
Alibaba's open source microservices will help developers and users microservices with service governance, and integrate cloud products to improve everyone's experience.
ACK+MSE = Cloud Native Architecture Upgrade Solution ASM+MSE = Service Mesh Solution AHAS + MSE = Microservice High Availability Solution ARMS + MSE = Microservice Observable SolutionEDAS + MSE = APaaS SolutionSAE + MSE = Microservice Serverless Solution WAF + Yundun + IDaaS + MSE = Microservice Security Solution
Operational activities
Limited Time Discount (4.21-4.30)
Microservice family bucket, province, province, province~
Upcoming Issue - Kubernetes Ingress Best Practices
With the popularization of Kubernetes, Ingress has become the traffic portal of cloud native architecture. As the best practice of Ingress, how can cloud native gateway help reduce business costs and improve efficiency, how to establish a secure and highly available line of defense from the portal, and how to achieve smooth cutting from Nginx Ingress To the cloud native gateway, 4.28 will be announced for everyone!
Aliyun MSE rush purchase entrance: https://www.aliyun.com/product/aliware/mse MSE international station purchase entrance:
https://www.alibabacloud.com/product/microservices-engine
Click here to watch videos related to microservice best practices~ Release the latest information on cloud native technology, gather the most comprehensive content of cloud native technology, regularly hold cloud native events, live broadcasts, and release Alibaba products and user best practices. Explore cloud-native technologies side by side with you and share the cloud-native content you need.
Follow the official account of [Alibaba Cloud Native] to get more real-time information about cloud native!