Transformation and communication-the best practice and development trend of microservice architecture (through case analysis)

Microservice

Readers may be technical backbones or technical business leaders in the enterprise, and they will further think about whether there are best practices to refer to and follow during the implementation of the microservice architecture.

Today I will share with you a summary of 1614, 1 Case, 6 principles, 1 method, and 4 development trends. What is a microservice? The champion made it very clear just now. Around the beginning of 2019, I wrote a book with a few friends called "Highly Available Scalable Microservice Architecture". My book also systematically explained what microservices are and what technologies are there. Features, how to do microservices around dubbo and spring cloud, etc.

Today I mainly share the book of microservices, some reference cases and best practice principles I summarized in practice, and my general judgment on the overall development trend of microservices.

• A Case - A Case Study of the Implementation of Microservices in the Financial Industry
• Six Principles-Six Best Practice Principles of Microservice Architecture
• One method-general process method of microservice architecture practice
• Four major trends-the development trend of microservice architecture

A Case - A Case Study of the Implementation of Microservices in the Financial Industry

Everyone knows that in our country, like the financial industry and other companies that strongly rely on IT for development, there are about three stages of IT development:

• The first stage is from 1999 to 2008, mainly from the traditional distributed accounting to the realization of electronic informatization process.
• The second stage is from 2008 to 2014. This stage is mainly network mobilization, which realizes that all of us can access the mobile network through mobile devices and use system services.
• The third stage is from 2014 to the present, which is a digital and intelligent process. In this process, user data is getting larger and larger, which in turn can promote the development of the entire business based on our data and realize technology-driven operations.

 

I myself have the honor to complete my university studies in the first ten years of the first stage and participate in the financial IT construction of the domestic banking industry. At this stage, I was fortunate to witness the process of many banks' transition from electronic to systematization. In the second stage, while doing banking matters, it also participated in Internet technologies such as Taobao, participated in some of their architecture projects, and participated in some network and mobile processes.

Looking at these three stages in general, with the development of IT systems, the amount of data and customers in the financial industry is increasing. The system of the entire financial industry is becoming more and more complex, and the non-functional requirements for the stability, consistency, availability, scalability, and maintainability of the system performance are also getting higher and higher.

 

Take a core financial case that I was in charge of before. When I took over, the system scale was relatively large, with more than 10 business lines operating on it, and probably hundreds of functional modules. The number of active users of this system is more than 1 million, and the total number of users is close to 100 million. About 200-300 million transaction orders are generated every day, which is about 10 times the number of daily transaction orders on Taobao and Jingdong.

The entire system's order processing capabilities, account and fund settlement capabilities are severely restricted, and the stability is very poor, often crashing. The transaction amount is also relatively large, with a daily transaction amount exceeding 10 billion US dollars. The system has very serious stability problems, the research and development efficiency is not high, more than a dozen BU business lines, noisy every day, so all parties are very dissatisfied, including customer satisfaction is relatively low.

Before I took over, the team also tried to do some splitting and microservice transformation. At that time, the scale was relatively small, there was no business involvement, and it was simple and rude, so the core system was torn down. Then I quickly found that I couldn't control it. On average, one person maintains 1.5 subsystems, and the stability of the entire system is not as good as before. In the process of transforming microservices, we first align all business operations, products, and technologies, determine the overall business centralization construction and how to transform the biggest goal, and disassemble the entire system into the front desk, middle desk, and back office.

The front desk is the financial front-end 2C business, which changes every day. This thing is not suitable for the core business as before. Like traditional banking business in the past, many small and medium-sized banks can't put their business at the core, so they put it on the front-end business. We have subdivided the system into foreground, middle, and background. The background is the infrastructure to ensure the minimum stability of the entire system. The middle station is also divided into three layers, the outermost is the API access layer, and the latter two layers are the business middle station and the technical middle station. These two are the core of the microservice. We have aggregated many basic business units, and the technology center is mainly service governance and service gateway.

 

At that time, the overall reliability of the infrastructure was probably only 2 9s, less than 3 9s, but what if our business system wants to reach 4 9s? We need to make big changes in the middle of the system, and use it to support the above without being affected by the underlying infrastructure. The security, compliance and stability of the entire financial system is the core part, so at that time we cited many strategies and did a lot of system stability construction. Fortunately, before the team, there were many Google and Amazon experts who did the construction of related solutions. Two years before I took over, they began to make stability-related preparations.

After taking over, I got all COE failure reports two or three years ago, how many ABC failures, how they happened, and the reasons for them. Through these things, all historical failures can be counted. Those systems, those people, and those teams, if a failure occurs, which link occurred? Through these to optimize and integrate the team, optimize our technology and optimize our process, many problems are fundamentally solved. Process problems belong to process, human problems belong to people, technology belong to technology, and management problems belong to management. After one piece is done, the instability factor is reduced by 99%.

 

The service-oriented things above need to be supported by a stable technical foundation, so we have consolidated the bottom layer, and we must have new and rapid ideas in infrastructure construction to solve the original problems.

• The first layer is the database. There is a period of time when the database is too large to be backed up, and there is always a long synchronization delay. If it is not handled properly at that time, many problems will occur. Then split the database multiple times vertically and horizontally to reduce the entire capacity before doing backups.
• The second layer is message middleware and memory middleware. The business is very sensitive to the low latency of transactions, but the previous core transactions were based on databases and disk files, resulting in high latency. Here, the transaction process is optimized by introducing message middleware, and at the same time, the original interaction of each node in the system is introduced into memory processing, and the final delay is reduced by an order of magnitude.
• Finally, the strategy of containerization and automation is introduced to reduce manual operations and gradually improve the team's technical maturity and R&D maturity. After all, the lesson learned before is that the R&D maturity of the entire system can't keep up, everyone is still in the traditional era, and you can't do something high-tech for everyone.

In this process, we introduced some data centers and data governance, and layered the data. Because the amount of data every day is too large, we divide the data into hot data, warm data, cold data, and ice data. Hot data is stored in a relational database, warm data is stored in a non-relational database, cold data is stored there first according to various requirements, and ice data is directly killed.

After the entire transformation, the throughput was increased by more than 50 times, and the delay was reduced from the original 600ms to less than 30ms. The maintenance cost of the entire system is rapidly reduced, and the time for business connection is reduced by more than 80%. In the past, it took two weeks to connect a business, and only two days after the system was processed. The stability of the entire system can basically meet our expected requirements, reaching 4 9s. After the transformation of the entire system, there was a qualitative leap.

 

Six Principles-Six Best Practice Principles of Microservice Architecture

In the process of leading my team to do microservices, I summarized six best practice principles for the transformation of microservice architecture:

1. How to deal with legacy systems?

The development of the entire infrastructure is from simple to complex. A system such as the microservice transformation previously done is a system that has developed to a certain stage. It is difficult to process at the business level, and the amount of data is huge. If there is a problem, it will be transformed only if Hold can't help it. For this kind of legacy system transformation, what is a better way to implement it? I sum up five forty words:

1. Function separation and data decoupling not only change the application system, but also split it into the business center according to functions and vertical capabilities. At the same time, the database needs to be split. I see that many R&D teams just deploy system application clusters and instead split several modules according to business capabilities, but the database is still a big pile. In this way, the database is not split, and all the pressure is still in the database, which is restricted by the coupling of the data itself.
2. Natural evolution, gradual dismantling, first dismantle the 10% and 20% that most affect stability or the most urgent business needs in the system, dismantle first. Regardless of whether it is cut into two pieces, or cut an unimportant small function first, it is actually a feasible way. But for this business demand, or stability demand, where the eyebrows are the hottest, the best results can be seen. It can also enhance the confidence of all parties involved in the subsequent renovation work.
3. Run in small steps and iterate quickly. Every time you split a small piece, this small piece can be maintained separately. In a previous case, there was a module that has been developed for many years. This module has 100,000 lines of code. It is found that the old system cannot be changed. It is okay to delete the problem. If the problem is not clear, let it be in it. Stay. We split it out and wrote it again, and we finished it quickly, using less than 8,000 lines of code.
4. It is also very important to publish in grayscale and be cautious about trial and error. Through the introduction of various publishing methods (rolling, grayscale, blue-green, canary, etc.), to ensure the stability and reliability of the release. We found that 80% of the failures occurred when the system was released or changed, just like an airplane accident occurred 5 minutes before takeoff and 5 minutes before landing.
5. Raise the quality line and pay off the technical debt. The quality of each system must be slowly reduced with long-term stable development. Every time we split a part, we can cut it out, clean it up, and see how it is done. In this way, the previous quality can be made up and the previous technical debt can be made up.

2. How to do the splitting with the proper granularity?

The principle of separation is: high cohesion, low coupling, and different points of separation at different stages. According to actual needs, it may need to be run-in in the middle after splitting. In the first major splitting cycle, we found that some things were unreasonably split and then added later.

3. How to expand the microservice system?

There is an extended cube concept here. All system extensions can be grouped into this cube, which has three directions, XYZ axis.

• The X axis is horizontal expansion, which is actually machine replication. If we add F5 and nginx in front, we can copy the business system brainlessly to N systems for load balancing and even out the pressure.
• The Y-axis is functional decoupling. We can split all the different modules of users, commodities, transactions, and payments into an independent system. The first step in the process of splitting is to expand the system. After the split, you can expand different system contents according to different system characteristics.
• The Z axis is the data partition, and the data must be divided. There are two meanings in this. The first is horizontal division, in which all data is split indiscriminately and partitioned. The second is that many times we find that the use of the entire data in the system is not even, it is inclined. For example, for the transactions between VIP users and large accounts in the system, we can make a separate machine for VIP to process his requests, and expand according to the characteristics and tags of the user data itself. This piece is also the basis of the unitized architecture we will talk about later. From this expansion, my experience is to strengthen the feature switch, so that the new and the old plus some fault-tolerant designs are compatible, so that after our system is online, if there is a problem, we can always switch according to the feature to reject the flow back to the old one. System, or gradually migrate traffic to the new system. After stabilizing for a period of time, it is stable and then the old one is dropped. There is a process of dual core coexistence.

 

4. How to improve R&D efficiency?

In the past, it was not easy to test a single system before the split, and it was easy to do deployment and operation and maintenance. After the split, the system became larger and more complicated. It is a big challenge for development, testing, deployment, and operation and maintenance. At this time, relatively strong experience in automated processing is required.

5. How to choose distributed transactions?

From XA supported by the database at the beginning, to TCC, SAGA, AT, etc. on the business side. TCC business is three independent small business, not one big business. These three separate and connected small transactions have high requirements for business design, which leads to the so-called business intrusion. At this stage, I see many items are used incorrectly.

The SAGA transaction mode is to directly do various cross-database services, and do the undo operations one by one when there is a problem. There is no distributed transaction solution. It is the same as the distributed transaction concept that everyone used many years ago. The business compensation/reconciliation model is the same, so at present, everyone in the financial industry uses the most handy.

AT is an automated generation of reverse operations and SQL, which is more intelligent, but it is not too mature at this stage, and complex logic may not be able to handle it. These types are all flexible transaction mechanisms that are controlled on the business side in addition to the database, also called eventual consistency transactions. In fact, relational databases and even some middleware such as MQ may have built-in support for distributed transactions such as XA with strong consistency. Since ten years ago, this technology has been used the most in financial transactions and other places where strong consistency is required. The champion teacher also mentioned a lot of commercial solutions and open source solutions for distributed transactions, so I won't say more here.

 

6. How to do monitoring and operation and maintenance?

Monitoring operation and maintenance involves the observability of microservices, which is also a very interesting topic. On the one hand, monitoring needs to support our business operations, on the other hand, it needs to cooperate with the technical infrastructure. In cooperation with the business, my biggest feeling before is that our monitoring should not only be done from technical and system indicators, but also from business indicators. Many times we find complaints from business departments and customers, what is wrong with the system, we see that all the system technical indicators are good, it is certainly not our problem. At this time, serious conflicts and chaos occurred between technology and business.

Then, the various operations done in our system will directly add the business indicators of the system, first look at the problem from the business indicators, and then look for clues from the technical indicators. At the same time, do capacity planning, alarm and early warning, and solidify the operation and maintenance process. We have found that if many technicians are responsible for and lead the launch, they will definitely think about solving all the problems that occurred during the launch process and let it be successfully launched once. However, our experience is that there are problems during the online process. The best way is to restore the version before the online, save the data at that time and then analyze it slowly. Do a good job of troubleshooting. There are natural and man-made disasters in our faults, and natural disasters cannot be avoided. The system is like a person. After a long period of time, problems will definitely occur. Therefore, it may be a normal phenomenon that various traffic abnormalities and crashes occur. We must accumulate sufficient solutions and experience to deal with it.

We often say that if we have an expected shutdown for maintenance, we tell all customers first, then this is a story. If we find a problem, do not stop for maintenance, and wait for a sudden downtime at a certain point in time, this is an accident.

 

One method-general process method of microservice architecture practice

Having said so many things about microservices, how do microservices work in specific projects? I have summarized a total of eight steps. The first four steps are the preparation phase, and the last four steps are the implementation phase. Then you can return to the preparation phase and continue to iterate the process:

• The first step is to research, first understand the current situation of the system and the demands of all parties
• The second step is to clarify the core issues and transformation goals
• The third step is to complete the transformation phase and plan according to the goal. It is unrealistic overnight, especially when the system involves thousands of people working together.
• The fourth step is to coordinate system-related teams and resources. The first four steps together are the preparation phase. The four steps of the preparation phase are different from the four steps of the implementation phase.

 

The next four steps are splitting, deployment, governance, and improvement. From technical monitoring and operation to the entire system and process, is everyone's cooperation OK? If there is a problem, no matter which part of the problem is to be improved, and then iteratively In this way, a large business system is gradually changed to microservices.

During this process, someone will ask me, would you recommend me at which point and what method should I use to deal with the problem? You tell him that you don't need to pursue the latest and greatest technology of the system, you just need to pursue the technology that matches the maturity of your current team, especially in the financial industry to be independent, controllable and open source.

Four major trends-the development trend of microservice architecture

Finally, let me introduce the development trend of microservices. From monolithic architecture to vertical architecture to SOA to microservices.

 

The concept of microservices is relatively early. It has been around since 2011. As everyone knows, it was in 2014. The country started to do the first wave of microservices in 2014 and 2015. In 2016, there were responsive microservices. Services are now more popular. Grid concept and cloud native concept.

 

The development of microservice architecture is in full swing, and there are currently four major development trends.

 

Trend 1: Responsive microservices

Responsive microservices originated from responsive declarations and responsive programming. Among them, instant responsiveness means that no matter what the system is, as long as there is no downtime, users must be provided with responsive services. To put it simply, we are more and more aware that the system is like a human being, and there will be problems such as headaches and fever. When a problem occurs, do we directly go down or provide some services under certain conditions. ability.

Resilience and elasticity. Elasticity means scalability and contraction. In addition, message-driven, the system is like people, we need to make a choice, you do not provide services at all, or now provide services within a limited capacity. The example I often give, a system has three levels in the protection and current limiting of the entire system:

• The first level is the so-called flow control. For example, we restrict it according to the number of times per unit time and the amount of data accessed, with weights.
• The second is service degradation. If you can't do all the business, I will leave it to everyone to get cash.
• The third is the overload protection of the entire system. This system has long been unable to handle it. A bank branch used to leave work at 5 pm, but now I leave work at 3 pm, which is also overload protection. During the process, the entire system is not downtime, and the operation of the minimum service is retained, which means that its controllable recovery is just like a human.

Trend 2: Service Grid and Cloud Native

In the past 2 years, Service Mesh is definitely a popular topic that everyone is familiar with. Different from the common Apache Dubbo or Spring Cloud microservice solutions, the service grid refers to the deployment of various non-business-related strategies in microservices from the business system to this node in a unified manner, removing the existing infrastructure from the process, that is Sidecar, microservices and microservices communicate through the local Sidecar, and Sidecar controls policy rules through the control panel, which is a microservice grid.

In this way, we divide the system into the data platform where the business services are located and the control plane where the control nodes are located. Therefore, in this new structure, the microservice node only needs to focus on the business logic itself.

 

Trend 3: Database Grid and Distributed Database

Similarly, when we apply these concepts to the database field, a new component is created, which we can call Sharding-Sidecar first. In this way, the business system does not need to manage the data protocol and where, but only needs to treat all the data as local data and apply to the Sharding-Sidecar component. All configurations related to the database itself, all data quality, security, flow control and governance strategies, do not require direct perception by the business system.

Correspondingly, we can also add Sharding-Sidecar to the data panel, and add our distributed database management and UI to the control panel. At the same time, we can use database nodes that are completely transparent to the business system, such as a batch of MySQL instances, as a new layer called the storage panel. At this time, a complete cloud-native database middleware solution is produced, which we call a database mesh.

 

Trend 4: Unitized Architecture

Many large organizations can’t directly do microservices in one step, so we can transform the original big pieces into mini versions in the process of microservice transformation, and even some parts happen to be self-consistent to form a closed loop and distributed in different Inside the data center.

 

The picture below is one of the four major state-owned banks that I recently participated in as an expert consultant, and their entire unitized structure. The PaaS layer (Tencent's TSF+TDSQL) used in the bottom layer should be known to everyone, and the upper layer uses our self-developed service management layer and unitized architecture layer.

 

Let's summarize the current development trend of microservices, from microservices to service grids to responsive microservices, to database grids and cloud natives, and then to some things we are doing now, to compare microservice architecture ideas and experiences with the current ones Combine large-scale financial industry business cases, such as modular architecture.

I hope it can be helpful for everyone to learn about microservices, and the friends I like can help with one-click three-connection, thank you!