本文内容源于是我在 2018 年北京 DevOps 国际峰会上的分享”成功的微服务应该是什么样”。PPT 可以在[这里下载] (https://pan.baidu.com/s/1jSleh_UxXpqI_oXOwcqf1w)
In the previous article, we introduced how to define the success of a microservice transformation, and introduced the characteristics of a successful microservice organization structure. In this article, we will introduce the technical characteristics of successful microservices and our experience in the implementation of microservices.
Technical characteristics of successful microservice landing
In many cases, the microservice cases we see are often a result, but lack of process. It makes us think that achieving the same structure result is even a success.
这是非常危险的想法，一个架构成功的关键一定是符合当前的组织结构并且让业务持续运转，无论是哪一种架构风格都不可能脱离这个基础。
Therefore, the following points are often the signs that we judge the success of microservices. However, the characteristics of these microservices will not necessarily bring you success. We must pay attention to measuring the scenarios and costs of adopting these technologies.
Hybrid microservice architecture
If I need to summarize a piece of technical experience, it is not to be superstitious about any technology brought by microservices. But through measurement, find the right technology for the current organization.
The figure below is an example of a microservice application, which used to be a large-scale system of multiple integrated monolithic applications. Generally speaking, we will make a layered architecture, we will make a layered architecture, with the UI at the top and the operating system at the bottom. These applications are hybrid architectures that use different programming languages and frameworks.
The microservice cases I have experienced do not have a 100% microservice architecture. They are all hybrid systems, that is, part of the system is microservice architecture. The ratio of microservices adoption depends on the input-output ratio of microservices.
We can make a simple estimate. For example, I can use three teams with a one-month budget, and then do this thing, how much output I can bring or how much cost I can accept.
You will find that some microservices are worth dismantling, and some microservices are not worth dismantling. After dismantling, it will increase the maintenance cost of the application system.
If you do not hesitate to pay, you must turn this architecture into a 100% microservice architecture. You have to understand these additional costs.
The architecture of the microservices we make may be microservices, and some may not be microservices. It is composed of multiple languages. Each application is a code base, so it is very clear to manage.
Many teams I have encountered who want to adopt microservices are often struggling with whether they are "is" a microservice architecture. And I think you can "have a microservice" in the architecture first and see the advantages and disadvantages it brings. Then consider how to extend your own microservices.
Microservice platform
The development stage of the distributed architecture of large-scale applications is shown in the following figure:
The earliest monolithic application (era1) is often a JSP application running on Java middleware. Our application was also like this at the time, and some JSPs were mixed in it. This is the architecture of most legacy systems.
Later, with the needs of mobile and distributed. We have done a front-end separation, using Restful API as the back-end to provide data interaction for PC browsers and mobile apps. In this way, there is no need to write a separate background application for the mobile application, just reuse the previously written API. This is the first step (era2) for many applications to transform microservices.
In the later stage, we found that some APIs need to be converted, so we made something called Syndication API, which is actually a collection of APIs. Re-expose the back-end data and interfaces through the reverse proxy. This was an excessive plan to be able to provide data to the Mobile terminal at that time.
Later, we discovered that to complete a function requires repeated development for the Web and Mobile terminals. Therefore, we decided to gradually replace the old web application on the basis of separation of the front and back ends, even if it runs stably.
In order to reduce the risk, we once again split the Syndication API. A single API set is divided into API groups according to the degree of dependence of LoB functions.
In this way, we can modify the back-end deployment architecture without the user's perception. At this time, although we did not achieve microservices, we separated the complete business through our respective APIs and created an adaptation layer between the legacy systems to isolate risks. In the future, we only need to write code to replace the original logic.
This architecture is one step further than the previous architecture. If you use cloud or infrastructure as code technology, you can automate the deployment of applications. This is how microservices first appeared.
Later, Docker appeared. That is, the third-generation microservice technology stack (era3) in the above figure. We use Docker to encapsulate applications in containers and expose Restful APIs uniformly. We will divide the front-end and back-end applications into containers to run, which will be convenient from the perspective of operation and maintenance.
But the risk of splitting the database is still relatively high, especially when your system is large and the database structure design is not very good.
By the fourth generation (era4), we will be a container platform. Use containers to deploy all applications. We will do the front-end on this platform, and there is only a little code in the back-end. We will run it on the platform, and all states are independent.
But note that a container platform only completes the life cycle of the container, and it is still a certain distance away from the life cycle of microservices.
Therefore, we also need to do life cycle management of microservices on the container platform, which is also the direction that companies that are now at the forefront of microservice transformation are working hard.
Independent storage/hybrid storage
Another characteristic of successful microservices is that the database can be split and expanded on demand so that you can maintain it independently.
But if your database performance is good enough or your database structure is not very good, you can keep this way.
Not that the split of the database is necessary. At the beginning, we tend to use the single database on the left side of the figure above, and the form of multi-microservice access. Later, we will split it into the form on the right.
When the database is split, attention should be paid to data redundancy and consistency. In order to improve efficiency, proper redundancy in independent databases is necessary.
However, if in order to avoid redundancy, and constantly cross-database, cross-API query. It is very likely that your microservices are disassembled incorrectly.
Splitting and reorganizing the database from the query frequency and performance of the database is also one of the techniques for splitting microservices.
The general principle is: "First dismantle the meter, then dismantle the library.
Associated queries are split first and then merged".
This will be an iterative calibration process, and it is difficult to succeed at once.
In addition, I prefer to write the relationship in the application logic instead of the database, so that you can reduce some of the underlying dependencies. Can isolate chain problems caused by centralized database management.
Lightweight inter-service communication
Communication between microservices is an unavoidable scenario. However, if the communication between your microservices is too frequent or the performance is low, you may need to reconsider whether it needs to be split.
At the earliest we used ESB to transmit data, but later we discovered that ESB would bring additional side effects.
Later, we replaced the ESB with RPC calls to implement message passing in the distributed system from the bottom.
At this time, we found that although RPC helped us solve some dependency problems, the relationship between system components became more complicated.
RPC is relatively low-level, and debugging is relatively difficult. In many cases, it will evolve into an external coupling.
Sometimes we also use database tables for message passing, which is actually similar to the database split we introduced above, and it will also become a kind of low-level coupling.
In terms of the coupling effect, the performance is the same as that of RPC, and sometimes there are some advantages of the database, and if RPC is not handled well, there will be blockage caused by synchronization.
Microservices tend to have a unified data communication format and asynchronous calls, which reduces the occurrence of blocking and improves the performance of the system.
So we replaced it with message queues and API calls. The message queue and Restful API Call itself are even an important way of system splitting.
It transforms the internal dependence of the system to the external. But even if the message queue and Restful API are used, there will be blocking.
What needs to be explained here is in terms of performance and throughput alone. Several methods may be similar, and even the underlying method is better.
But the point here is that we expose internal dependencies to the outside through message queues and Restful API, so that internal "dark knowledge" can be fully exposed. Reduce the risk and repair time of the system. From this perspective, the throughput and stability of the "file interface" is also a solution that can be adopted.
And if your microservices communicate too frequently and bring additional risks and costs, it is possible that your microservices have been disassembled incorrectly.
In this case, you may need to consider merging microservices. This depends on the performance and stability of the system, and also on your maintenance costs.
A successful microservice split will have fewer dependencies between services. Along with this, the number of synchronous calls in the application system has decreased, and the number of asynchronous calls has increased. Replacing synchronous calls with asynchronous calls can also be seen as a way to split microservices.
On the other hand, a successful microservice split also brings development independence. Since your service is invoked asynchronously, it should be able to be deployed and released independently, without relying on other processes. If your microservices are split and the development process is still very long, consider the split in organizational processes.
Microservice full link monitoring
In the picture above, we use NewRelic to monitor the microservices. You can see some business systems on the left, then the microservices in the middle, and finally the database on the right. For confidentiality, I covered the name of the microservice with a blue box, as you can see. This is a mixed architecture of NodeJS + PHP and MySQL.
Your microservice operation and maintenance also need to have a connection diagram for the health check of the microservices. In the past, you may only need to focus on the performance of one application. After arriving at the microservice architecture, you need to pay attention to every node.
只有一条依赖上的所有服务都健康了才算健康，但是中间如果有一个不可用的话，哪怕它的失败率很低，如果你的依赖链很长的话，你的应用的健康度就是这些节点的可用率相乘的结果。
For example, if you have three dependent microservices, their availability rate is 99%, then the business availability rate is 99% 99% 99% = 0.970299, which is 97.02%. But if the three microservices do not depend on each other, their availability rate is still 99%. Therefore, the more dependencies between services, the higher the risk of the system. This also proves the advantages of microservices from a mathematical point of view.
of course. We cannot consider the success of a single application when we consider it. What we consider is the overall monitoring of the failure rate of this cluster to obtain the availability rate of system monitoring.
When the original system's internal dependencies are exposed to the outside, the operation and maintenance work is not only concerned with the previous "big black box", which requires the joint cooperation of development and operation and maintenance. This is also the main reason why the microservice team must be the DevOps team.
The easiest way for us to do microservices is generally to do input-output analysis first, and then maybe do the separation of front and back ends, and achieve continuous delivery of front and back ends.
The point here is that infrastructure governance needs to be done in an automated way. This is difficult without DevOps organization.
Our usual practice is to develop a new set of design for microservices and separate the microservice team separately, because these are two different cultures and processes. Finally, a full-featured DevOps team and microservice template are achieved. The automation of these infrastructures can be replicated.
We can continue to replicate successful microservices in this way, and we have the success described above.
Reflection on the landing of microservices
To sum up so much, the above are just my observations in the microservice transformation projects I have experienced. However, everything is not as beautiful as it seems. We have also taken many detours and stepped on many pits in the process of obtaining the above results. Let me share these experiences with you.
Split microservices on demand
It took 5 years for our customers to advance the microservices of the overall system to a rate of about 60% to 70%, and they will not continue. It is because there is no reason for further microservices.
On the one hand, the investment income of further microservices is not significant, on the other hand, because other systems are either integrated by corresponding vendors or are preparing to migrate to SaaS-based applications.
In any case, it is to reduce the risk of the application system and reduce the cost of maintaining the normal operation of the system.
As I said before, when you want to do microservices, don't rush you to "be" a microservice architecture. Instead, there should be a microservice in your system. Then take a look at what kind of effect this microservice brings, do a good job in all aspects of measurement, and what is its investment? What are the benefits? Do you need to spend more to manage the middle problem? And you have to face some uncertain risks?
So, if you start to do microservices, you can consider setting up a small microservice team first, and then get a feel for it.
Consolidation of microservices
My experience is if you start to merge microservices. Show that your microservices practice has reached a maturity level. You start to think about when to dismantle and when to close.
We once had a subsystem, which was assigned to different departments in two or three changes due to organizational changes. After finishing the demarcation, our microservices were disassembled, closed and dismantled, until the last dismantlement, and then dismantled and dismantled 3 times.
Keep in mind Conway's theorem when doing microservice disassembly and reorganization:
There is a one-to-one correspondence between your system structure and your organizational structure. In this way, you can find problems from a macro perspective, and also know which ones are insurmountable "sound barriers."
Another situation is due to performance or to ensure the integrity of the transaction, which is often due to the deployment architecture is not considered in the design process. Therefore, in order to avoid these problems, it is very important to understand and plan the deployment structure in advance.
Hexagonal architecture
Can you draw your architecture diagram using the hexagonal architecture? In domain-driven design, the "hexagonal architecture" is also called the "adapter-port" model. If your team can use the hexagonal architecture to draw your architecture diagram when changing the architecture diagram, then you are not far from microservices, but you have not realized or automated deployment.
Align "UBIQUITOUS LANGUAGE"
The next case I want to talk about is the alignment domain language. Our clients do search engines and online advertising, and its source of income is advertising.
In foreign countries, for customers, the price is tax-inclusive. For internal calculation statistics, the quotation is the net income excluding tax. This ratio is about 10%. However, since we all use the unified word Price as the field for exchanging data between systems. Therefore, the system exposes the internal tax-free price to the customer.
When we discovered this problem, the system had been running for three years. In other words, this system has lost 10% of its revenue every year for the past three years to help customers pay taxes.
The whole process is because when we added automated integration test cases to these systems, we found that the automated test failed, and then we found the price inconsistency between the two systems. Although they are all prices (Price), one includes tax. , One excluding tax.
When you go to integrate with different systems, you will find that aligning the unified language is very important. This is also a problem we will find in the process of doing microservice architecture. Because programmers don't understand business, it will cause such problems. So you need to have a domain expert to align domain languages when integrating different systems.
Later, we expressed the word "price" in two different words in different systems. In this way, we know that we need to reconsider whether it includes tax when calculating the price.
Then, we wrote some automated tests to ensure that the calculations are correct. Once the tax rate is modified or the rules are modified, we know where to modify the corresponding business logic.
Don't develop unified tools prematurely
The tool of unification is a double-edged sword.
As a programmer, I always like to keep repeating the wheel to automate, and then I will make some tools to speed up my work efficiency. But when you make a unified tool, you will require every team to use this tool, which will create new dependencies.
Moreover, this tool may not have mature open source solutions. If you want to develop a tool, you need to invest a person or even a team to maintain it, it is likely to enter the "golden hammer " anti-pattern.
In the early days of microservices, when there were few common scene recognitions, its scenes were limited. Once the scene changes, the unified tool becomes an obstacle. When the following scenes become more and more complex, you need to constantly modify the tools to ensure backward compatibility while ensuring forward compatibility.
You may find that your time to develop microservices is much less than your time to deal with the problem of unified tools, and you have to spend extra costs.
Our customer has developed a lifecycle management tool that combines AWS to publish Docker applications and the corresponding microservice template.
But as the scene increases, this tool becomes more and more complex. Small deployment requirements require the addition of many useless components and configurations.
So I removed this tool and used the simplest AWS native way to deploy the application.
If you make a unified tool, then you promote it as open source. If you make a tool and don't promote it, your tool will be replaced by other mature open source tools, and your R&D investment will be wasted.
If your open source effect is good, maybe it becomes a certain standard in the industry, there will be many people helping you to maintain these tools.
I have heard a story from a major Internet e-commerce company: Due to the need for Dockerization, it developed a set of very advanced container management platform, but it was held in its own hands and not open source.
The later story is that they replaced their hard-developed container management platform with Kubernetes .
So when you start to make this tool, you must consider its R&D cost and replacement cost. One way to reduce this cost is to open source.
On a global scale, it is meaningless for us to write software to construct technical barriers, because there will always be someone who will open source a good solution. Then it becomes the object that everyone competes to imitate, and then the de facto standard is formed.
In this case, if your team wants to develop a unified tool, please first consider whether there are mature open source solutions in the industry. The cost of making wheels, unloading wheels, and changing wheels are all high. The software industry has been developing for so many years, and there is nothing new in the world.
Growing thick microservice platform
Another aspect of unified tools is the microservice platform. Our customers have established a "PandA team" (PandA, Platform AND Architecture platform and architecture team) to lower the barriers to deployment and release of microservices. This is often due to the "second split" of DevOps.
When the application of DevOps becomes more and more widespread, the boundary between development and operation and maintenance has become clear once again. Make the development and application faster and faster, more and more standardized. On the other hand, the operation and maintenance department will SaaS all services and provide unified specifications to reduce management costs.
Then comes another problem: when we use processes, tools, and apply more platform concepts. We will find that the entire workflow may no longer be unagile. In agile, we say that individuals and interactions are higher than processes and tools. In this case, we will add more and more processes and tools, thereby reducing individuals and interactions.
When we emphasized agile earlier, we thought that the left and right terms of the agile manifesto were opposites. But now we review the Agile Manifesto, can we have both at the same time? It not only promotes the individual and interaction, but also has processes and tools. When thinking from this perspective, we will create new tools and new methodology to resolve its contradictions.
This is what I call the post-DevOps era. Our operation and maintenance team has become an internal platform product team, which is providing an infrastructure product to the development team. The development team has become an external application product team, which is providing users with a product that meets business needs. Independent of each other while maintaining the integrity of the DevOps life cycle.
At last
The above is part of my observations on this customer's microservice transformation over the past five years. As a personal experience and witness of microservice transformation, I was fortunate to observe how an organization changes from the inside out through microservices.
In my experience, microservices are the inevitable result of the in-depth DevOps. When our deployment and release encounter bottlenecks, we need to separate concerns and risk points, and adjust the application architecture to further improve DevOps feedback.
When we start to practice microservices, we must first unify the understanding of microservices, so that everyone has a unified understanding of microservices.
The second is to understand why we do microservices? Microservices solve your problem, instead of falling into blind technology worship.