This paper finishing from GIAC (GLOBAL INTERNET ARCHITECTURE CONFERENCE) Assembly of the global Internet infrastructure, ants gold dress platform data technology business group of technical experts Shi Jianwei (nickname: Zhuo and) a share. Share based on the concept of Service Mesh, combined with ants gold dress inside the actual scene, the ability middleware, data layer, security layer such as peeling from the application out after sinking to a separate Sidecar SOFAMosn, the combined Kubernetes operation and maintenance system, offer seamless case upgrade infrastructure layer capability in case of perception.
The share will be expanded in the following order:
The current status of the service of ants gold dress
Before looking at the service of architecture ants gold dress Let's start with a simple example of service of the call talking about, below is SOFARPC basic principles:
We can see from the chart, to build a service oriented framework requires registration service center, service definition, the caller and service providers use the same service definitions to communicate with each other. Through a service registry, the caller can subscribe directly to the address of the service provider, using point to point the way to make requests directly. It can be achieved within the client service discovery, routing and load balancing, limiting the ability to fuse and other communications capabilities to enhance service. Through our open source SOFARPC, SOFARegistry, SOFABoot, users can already built up a micro direct service system, boost business development.
Ants gold service development to date, 11 dual system needs to deal with peak trading on a yearly basis:
265,000 transactions per second peak data 2017 double 11, the data behind a very complex architecture support, LDC unit architecture ants gold dress sediment core architecture for many years, relying on this architecture to achieve peak annual trade volume growing rapidly under The system can still smooth ride. Let's briefly look at LDC architecture:
The figure from "financial level distributed architecture" in the sketch unit of a text, not to proceed here in detail. LDC cell architecture to a service oriented applications bring more abstract norms, service routing calls between cells need to be considered, across more room scene recalls. The main expression of this hope is RPC calls to LDC architecture brings greater complexity.
Services of pain points
Middleware upgrade
In presenting the above background, it is introduced to the complexity of the current service call at LDC architecture, the complexity is directly reflected in the current application code. For business students is concerned, the focus is how to achieve a business application logic, as high availability, disaster recovery and other capabilities are more points overall architecture level will be considered. At the same time through the introduction of RPC jar package to get the next service call LDC infrastructure support capabilities within the application, the convenience can also see the shortcomings of this model:
In addition to the business logic of the application, the middleware SDK large external dependencies introduced to complete service discovery, routing and load balancing, the current limiting fuse, serialization, and other communications capabilities, the introduction of each component may bring stability risk, and higher upgrade costs.
According to the current version SOFARPC inside example, service discovery client do SOFARegistry dependent service within the IDC found, depend IDC Antvip do cross-service discovery, ZoneClient LDC information integration unit architecture, according to the routing address request into parameters required currently Zone then calculated to determine the call target, current-limiting fuse Guardian dependent components, communication protocols and serialization protocol is relatively stable, changing little. Only a complete service calls, applications need to introduce additional 7+ client package.
When it comes to annual double-11 architecture require adjustments: for example, supports flexible framework, need to do a lot of middleware upgrade clients to support better framework for business students is concerned, these upgrades are very exhausting, take the 200 core applications For example, each application middleware upgrade version through research and development, testing, deployment and then to advance, gray, production environment requires 5 man-days, then 200 core application middleware upgrade takes 1000 man-days, if this can be part-time freed up, infrastructure upgrades every year can save a lot of human resources.
Cross-language communication
Ants gold dress development so far, internal business flourishing, search recommendation, artificial intelligence, security and other services to use the technology stack is very diverse, cross-language communication ability of service is also very important. Some years ago, is the largest outside of Java NodeJS application, it can be reused in order to allow a variety of middleware and infrastructure inside the ants gold dress, front-end team NodeJS between Java and the applications that use a variety of NodeJS gradually rewrite middleware client, so that the whole system can be perfect NodeJS and Java interoperability.
Middleware SDK cross-language rewrite and high maintenance costs, with the increase in language, cross-language communication demands more and more.
Java, NodeJS, Go, Python, C ++, etc., + 5 languages, middleware SDK rewrite all costs extremely high. These issues have to inspire us to find better solutions.
Solve pain points
SDK ability to sink
In the above example is still the RPC SDK, the SDK ability according to characteristics of stability and we can not look at peeling, which can be drawn from the application, and try to do the SDK thin, stable enough to do without changing, the upgrade costs It will cease to exist.
RPC SDK service discovery, routing and current-limiting fuse and other features, is easier to change, this part of our ability to sink to a separate Sidecar, you can reuse this part of the ability to make multi-language SDK only achieve the most basic serialization and communication protocols, and these capabilities are very lightweight and easy to implement. Sidecar where we want it to exist as a separate process, and the process of stripping business applications, and business applications and upgrades decoupling open, enabling business and infrastructure development in parallel, without disturbing each other's vision.
In addition to RPC communication, we can also sink news, data sources, and other capabilities to Sidecar, a business application can be thinner, SDK implementation cost is also reduced to an acceptable level, the infrastructure layer and divestitures, mutually independent evolution .
Floor architecture
Overall structure
Istio different from the open-source system, ants gold dress internal version of Service Mesh landing priority to achieve and landing data plane, control plane and gradually build overall architecture point of view, we use the data directly and face a variety of middleware internal services butt end, the ability to complete the sinking of RPC, messages, etc., to business applications burdens. As can be seen from the above chart, we Sidecar with different business applications in the same arrangement in a Pod, App and SOFAMosn direct communication, SOFAMosn responsible for the targeted service interface discovery, route addressing, and built-in security module by the SOFAMosn do encrypted authentication between application calls. Sidecar DBMesh achieved by a data sink layer, App does not need to establish a connection with a plurality of data sources, need only connect DBMesh Pod in the present data layer to complete calls, the database user name, password, sub-sub-table rule library etc no longer need to be concerned.
Figure Term analysis:
ConfigServer: distribution center, responsible for various metadata profile, like the dynamic switch
Registry: service registry, responsible for internal IDC service discovery
AntVip: DNS resolution class product that can be resolved through a set of target domain address for service discovery across IDC
MQ: message center server configured to send and receive messages
Landing data
Currently this core architecture has been done in the payment link in the pilot, 618 large pro-Mesh Mesh application no comparison application CPU consumption grew 1.7%, growth in a single transaction as a whole takes 5ms. CPU increase is due to more of a process, after a request for an increase, RT will be steady slight increase, but these costs compared to the dividend brings the overall architecture, minimal, and optimized for the entire duration of the data plane is expected to gradually reduce resource occupancy, improve resource utilization.
Reduce the degree of bother
Middleware ability to sink in architecture point of view is possible, the actual landing of how to do no bother to change the wheels on the train running, low-disturb is a very important point to consider. By means of Kubernetes of good practice, the service container with Sidecar container arrangement is the same in a Pod reasonable architecture, Sidecar not interfere with the container business, the upgrade can be done without a sense of both sides to each other.
We decided to get business applications such as silky smooth as possible upgrade, mainly made the following optimization:
1. No sense mirroring
Kim served inside the ants also part of the application is not complete mirroring, or by way of the rich container to use mirroring, this approach is also in the process of being phased out, in order to make business more smooth mirrored, PAAS platform linkage throughout R & D system, the application process by way of packaging automatically generated to help the user complete Dockerfile active mirrored. Users do not perceive the mirror transformation.
2. Low sense of Mesh
After the image is complete, you want the model to be applied Pod container and container SOFAMosn deployed together, we need to underlying resource management and scheduling system to replace all the Kubernetes aid. Then a Mesh identification of applications increase in PAAS, which are identified by identification Kubernetes active and implantation of SOFAMosn sidecar accomplished Mesh access applications.
3. No sense Sidecar upgrade
After Sidecar and business process independent, there is a core demand is hoped Sidecar can be upgraded independently, in order to allow Sidecar try to upgrade the production do not bother, we achieved SOFAMosn smooth upgrade of Operator, by connecting to a running SOFAMosn migration realize the entire upgrade process completely aware applications. Of course, here bread his mouth a lot of challenges, we will detail later.
Floor Challenge
Sidecar smooth upgrade
At present we have achieved SOFAMosn smooth upgrade capability, the ability to SOFAMosn is mainly Agent RPC communication and information, the purpose is to smooth the upgrade process does not restart App business, business interruption request does not complete the upgrade version of SOFAMosn.
Since the application is independent SOFAMosn Container, as described in FIG, SOFAMosn need to do is to upgrade the thermal image within the Pod Alternatively, this heat must be replaced to protect the ability points:
1, Pod will not be rebuilt, application processes always up and running
2, replace the mirror will not affect the operation of the flow
3, the replacement of the entire image needs to be updated description Pod
In order to achieve these objectives, we have to complete the upgrade process by the above transformation of Kubernetes and custom Operator. Process is as follows:
Without taking into account among multiple mirror smooth upgrade to do a scene, through the process of Fork, FD can inherit the parent process to complete the migration of long connection, non-destructive thermal upgrading. After SOFAMosn run in mirrored manner, greatly increase the difficulty of a smooth upgrade. The whole smooth upgrade of difficulty is how to make SOFAMosn process in different containers may perceive each other and complete the connection migration.
Here we look at SOFAMosn upgrade process, how to protect the flow of non-destructive:
SOFAMosn normal flow process is divided into an inlet flow and an outlet flow, and can be seen as SOFAMosn Client deployment service provider within the same target in the Pod App, Server requests may be invoked, it may be one App App may be invoked side SOFAMosn. When the sum of request sent from the Client to the Server, will go through two intermediate connection length: TCP1 and TCP2, SOFAMosn ConnectionID records corresponding to this request, and initiate a response to complete the request.
When a new container is started Mosn, Mosn will attempt local Domain Socket according to a transmission request, when there is another Mosn process, Mosn V1 and enters the hot Mosn V2 upgrade mode, Mosn V1 will FD connection already exists and has been read data packet to Mosn V2 while V1 will not read data from the migrated FD, FD migration is complete for all traffic will be handled directly by the Mosn V2 Domain Socket.
After Mosn V1 and Mosn V2 into the hot upgrade mode, there may be circumstances after the request has been sent to Mosn V1 Server Server has not had time to respond, this scenario Mosn V1 to V2 when the migration FD Mosn, Mosn V2 will take over in FD after the return to the ConnectionID Mosn V1, when the received response Server Mosn V1 returned, this would request to Mosn V2 by Domain Socket, and to find connection Mosn V2 according ConnectionId TCP1, and in response to the Client.
Extreme Performance Optimization
SOFAMosn core network model is completely self-realization, we have done a lot of optimization across the network layer, we have multi-pen merger request by golang of writev to write once, reduce call sys.call enhance the overall performance and throughput . Writev while in use in the process, found that there are golang to achieve writev bug, cause some memory can not be recycled, we submitted to the golang PR fix this problem, has been accepted: https: //github.com/golang/go/ pull / 32138
There are many other optimization point, not going to describe in detail, reference purposes only ideas, such as:
SOFAMosn asynchronous log, and to prevent disk issues affecting the forwarding performance of SOFAMosn
Route routing result cache, space for time, improve throughput
Nearly 90% of multiplexed memory to avoid copying bytes
Cluster lazy loading, enhance boot speed SOFAMosn
Protocol layer optimization, low specific protocol version upgrades, reducing costs unpack
Evolution direction
Service Mesh control surfaces of the building we are also planning to move forward step by step, hoping to unified interaction model data plane and control plane, control plane try to follow SMI standards, increase support for the TCP protocol description, and gradually increase as the data plane SOFAMosn stability, reduce the frequency of change, with a more general model to describe service discovery, service governance scene, DBMesh based XDS will do the next configuration hair, hair unified and control channel data at face shut. Here the control plane interface directly middleware server is based on performance and capacity considerations, internal ants gold dress room service discovery data for a single million level, the use Kubernetes ETCD can not carry such a huge amount of data and control plane needs to do a bridge and serve different slice plane data.
Then there will be a complete product layer construction, by means of ideological Mesh architecture, ability hope to get more bonus infrastructure upgrades by sinking to the Sidecar way. Mesh layer of products will include a variety of surface morphology Metrics Mesh data acquisition, monitoring the market, ways to face common external control, shielding the complexity of the interaction with external systems Mesh, unified control plane and data plane protocol interaction, through the improvement of transport construction and maintenance system, enhance the ability to upgrade Sidecar gray under the Mesh mode, do not bother to use and stable unattended aim to complete the upgrade version.
to sum up
To summarize the full text of landing practice in the field of Service Mesh can be summarized as the following six:
• The application layer with the infrastructure layer separation
• Set a base layer write multiple reuse
• the ability to translate the data plane by landing priority
• reduce the degree of bother to enhance the efficiency of landing
• building a unified control plane configuration data plane model
• Performance, stability, observability continuous optimization
This article describes the solution ants gold dress in Service Mesh floor of evolution and the associated pain points, hoping to bring some evolution of thinking is different from the mainstream community programs for our readers through practical experience.
Related open source components mentioned address:
• SOFAMosn：https://github.com/sofastack/sofa-mosn
• SOFARPC：https://github.com/sofastack/sofa-rpc
• SOFARegistry：https://github.com/sofastack/sofa-registry
• SOFABoot：https://github.com/sofastack/sofa-boot