本文内容源于我在 2018 年北京 DevOps 国际峰会上的分享 “成功的微服务应该是什么样”。PPT 可以在[这里下载] (https://pan.baidu.com/s/1jSleh_UxXpqI_oXOwcqf1w)
Preface
At the GOPS Conference in Shenzhen in April, I shared the "Difficulties of Landing Microservices and How to Land Microservices Efficiently". This is a summary of the project I started in April 2017 and later published on my blog and "ThoughtWorks Insights" .
The case introduced this time comes from a client I served when I joined ThoughtWorks in 2013.
At that time, we were doing the transformation of the legacy Java system, changing the way part of this Java application was called from within the system from ESB to HTTP external calls made with Sinatra (a Restful API framework of Ruby).
At the time, I didn't know what we were doing was microservices, but I just felt that the application was decoupled by automating continuous delivery. The complexity of the system is reduced, the code is reduced, and the failure of the production environment and the progress block caused by the dependence during the development process are reduced.
At that time, the project was carried out for 8 months. In addition to satisfying the decoupling of the monoblock system, we also added functions and did some maintenance daily work.
At that time, we did not have an independent Ops team, and all operations and maintenance related work were done within the team.
We used Docker for deployment in 2014, which was a very novel thing at the time. Since we couldn't find relevant materials on the Internet, we wrote some orchestration tools ourselves to deploy Docker to the production environment... Of course, it was not clear what DevOps was at the time.
In the second quarter of 2014, we started to do contract testing, which is a consumer-side contract test. It is a self-developed tool, Pact, which is not developed by us, and we put forward a contract testing theory.
We will turn the serial integration into a quasi-parallel integration. You will see this tool in books and related practices on microservices, large and small, and they also define what contract testing is.
It was not until the end of 2014 that we had completed the whole thing before we knew what microservices are and what DevOps is. Later, this microservice project became one of the microservice cases that ThoughtWorks publicized.
In November 2014, I left this team and started to promote the experience on this team to different customers, only to realize that our own technical practice is already at the forefront, and only then did I gradually understand the concepts of DevOps and microservices. .
During this time, this customer also began to copy our previous successful experience and began a comprehensive micro-service transformation. Sam Newman, the author of "Building Microservice", also wrote our successful experience in this book.
By the middle of 2016, he said that we should do SRE instead of DevOps.
There are fewer and fewer functions they can think of. What they do is to integrate all the IT systems in the enterprise. The biggest problem at this time is the problem of system integration. We will also use the case of microservices to solve it later.
By 2017, they were growing very fast, from No. 1 in a country to No. 1 in the southern hemisphere. They made a Panda Team and took all the resources together to make a platform, and then we were on the platform. It can connect with products of the same type in different regions and countries, because they have already started M&A, and can connect with products of the same type in different countries and regions.
We know that the biggest benefit of microservices is that the same platform connects heterogeneous languages and heterogeneous products, so that we can quickly integrate them to help us develop.
In 2017, I returned to work on this client’s project and discovered that it was just 2 years after I left the team, which allowed me to compare the effects of microservice transformation from a new perspective.
This article uses this customer as a case study and discusses what a successful microservice should look like from four aspects:
-
What kind of microservice transformation is successful?
-
What is the organization of the successful microservices?
-
What are the technical characteristics of successful microservices?
- Key experience in the process of practice and landing
这篇文章里，我们先谈前两个问题。下篇文章里，我们谈后两个问题。
What kind of microservice transformation is successful?
There are too many micro-service legends on the Internet, and few have experienced it personally, and most of the cases we have come into contact with are from foreign countries.
We can only see a variety of new tools, but it is difficult to know how they use these tools.
Therefore, most domestic microservice practices seem to be the difference between buyer show and seller show.
What we see is often a result, but it is difficult to see a process.
Most of our experience comes from "what is a successful microservice?", but it rarely involves "what is a failed microservice?", "how to avoid failure?" and "what if it fails?".
Therefore, at the beginning of this article, we first define what kind of microservice transformation is successful.
但是这不是说你的微服务，按照本文所介绍的几点去做就是成功的，而是说如果你去做微服务如果成功了，它应该和这些点所带来的效果差不多。
After all, each system has its own specific domain problems and technical environment, and the path of implementation is quite different.
Our main appeal for microservices is to hope that while the scale of the system grows, management costs are reduced.
This management cost includes two aspects:
-
Staff management costs are reduced.
- The management cost of technology is reduced.
Feature 1: Reduced personnel management costs
As the business expands, more requirements need to be developed and maintained. Therefore, in the case of the same time and quality, the only way to meet the demand is to increase resources.
However, the law of diminishing marginal returns tells us that under other conditions unchanged, the return of any single factor input will diminish.
Therefore, additional factors need to be added to obtain growth and rate of return, especially to adjust the structure of the organization and reduce to increase capacity.
In the face of large-scale application system development, we only invest in people, so we can see:
With the increase of personnel in large-scale projects, the management costs it brings will also increase, until the increase in personnel does not generate benefits. This is a problem we see in large-scale application systems, especially Internet applications.
However, the emergence of microservices. Changed the application architecture, more precisely, changed the organizational form and reduced management costs.
The previous model of extensively decomposing complex issues and eliminating demand through crowd tactics is not sustainable.
The first reason is that with the explosion of requirements, the application system will become more complex, including external and internal requirements, functional requirements and non-functional requirements.
The second reason is that the cost of obtaining engineers who can reduce the complexity of these methodology, including the cost of recruitment and training, is very high.
The microservice architecture can greatly reduce the complexity of the application system and reduce the difficulty of management through the rapid replication of the agile team.
Therefore, we can see the team's self-improvement and the improvement of personnel capabilities, and you can have fewer managements. The case sharing of the Huawei DevCloud team on the 2018 Beijing DevOpsDays also proved this.
If your microservices do the best, according to Conway's law, your architecture will be consistent with your organizational structure.
If you have an autonomous and self-growth team, by turning best practices into systems and rules to fully authorize, without requiring more instructions and reports, your organizational structure will not have a very thick and high level. And the corresponding return relationship.
It is a flat and loose structure. Each team completes tasks independently according to the same system, and there will be no blockage in organizational processes. But it can achieve the same quality efficiency.
In addition, we have improved the speed of feedback through DevOps practices and reduced manual intervention through automated methods. One of the DevOps culture is to automate. Before, you would find ways to automate if you didn't know how to automate. You think that teams that open source software can't satisfy will find ways to do it.
And such a culture will eventually form a software development culture and system, which can be replicated and promoted within the organization. Reduce more labor management costs.
Feature 2: Reduced management cost of technology
Another characteristic of the success of microservices is the reduction of technical management costs. This has two aspects of costs. On the one hand, it is the development cost, and on the other is the maintenance cost.
First, you will get a clear architecture. The application architecture of most of our systems is chaotic, and there is too much "dark knowledge" in it.
"Dark knowledge" is the uncertain knowledge that needs to be monitored through tracking code, tracking log, and actual operation. Now let's draw a hierarchical diagram of the system architecture. The diagram you drew does not correspond to the actual system very much, and it is very confusing. It is often the case that the deployment architecture and the logical architecture are inconsistent.
Second, you will get faster and more stable release feedback. This is actually the benefit of DevOps. This allows you to deploy applications more frequently, and the applications are more stable during deployment and update, and there is less downtime.
In this process, you will find a lot of automation, and then do more and more with the microservices. The biggest problem is the management of microservices. In fact, the size of microservices determines the number of microservices when the external demand function remains unchanged.
It also determines the number of teams and the complexity of management. Therefore, the size of microservices is not a problem, the question is how many people you can support such complexity.
When you find that you need to manage so many microservices and you need additional automation, your team will drive a sense of automation.
We know that if your development tasks are increasing, the best way we can do without increasing capital investment and development time is to reduce the need to meet your final delivery time point.
But in the environment of microservices, you don't need to have such tasks to be delivered at due time. You find that you can solve these problems without adding more people;
Another point is faster business response speed. My business is 3 months on average, and now we have reduced it to 1 month. For the first service we did at the time, we could only measure that team. We now have teams of different sizes. Small requirements may last for two days, and large requirements may last for three months, but the results are different.
Organizational characteristics of successful microservices landing
Next, let's talk about the organizational structure of the successful implementation of microservices, and look at it from the perspective of organizational structure. From the end of 2014 I left this client's project, to the beginning of 2017 I returned to this client again. I didn't work on this client's project for the middle three years, so I can see the changes before and after the adoption of microservices.
Multiple microservice teams
From a macro perspective, an enterprise must have many different application systems in order to meet all aspects of informatization needs. Such as finance, personnel management, product management, work flow, etc. When the development reaches a certain stage, it will be necessary to realize data sharing between different systems through technical means.
We will use system integration technology to integrate different systems and integrate all systems together. Two issues are involved here:
One is "Single source of truth", which is a single source of truth. We hope that in the case of multi-system integration, a certain kind of data, such as customer information, prices, etc., have a single source of truth. Otherwise, there will be data inconsistencies between different subsystems.
The other is the Design For Failure mentioned earlier. During the operation of the business, the transformation of the application system cannot make the current business collapse.
Therefore, any one of our decisions must maintain the stability of the existing business operations, on the one hand, the personnel organization, on the other hand, the system architecture.
The three colors in the picture represent three business systems. At the beginning of the three business systems, only Team A was used for microservices, and it was only a small part of an application, such as one of the microservices of APP-1. Other teams are still maintaining their own monolithic applications.
They split all application businesses into different microservices and integrated them, which took three to five years. The maintenance workload faced by their team seems to be greater, because they need to focus on more points, but its team has not increased but decreased.
Some teams were broken up and integrated with other teams. Or a new business unit has been developed.
There were a total of 120 developers in this company before, including ours and customers, but now there are only 80 people left. In the past four to five years, nearly 30% of people have left their jobs to start a Bitcoin business, and of course others have added in.
However, their system did not break because of the maintenance of so many modules, but so many people are enough. At the beginning, we had an operation and maintenance team, and the first microservice team worked with this team.
Later, it will no longer work on each team, but will form an operation and maintenance model. This team is the Panda team (PandA, Platform AND Architecture platform and architecture team) mentioned earlier.
What is the microservice team?
What size of microservice team is appropriate? Below is a photo of our microservice team. Amazon proposes two pizza teams. We have also used two Pizza Hut pizza teams, but we found that the two pizza teams were not realistic. It is because the granularity of the microservices you encounter is not the same.
Therefore, we formed a "two-table room" team, as shown in the figure below
The two pizza teams are not very scientific, and the strength of the problems you encounter is not the same. These are two tables, and the corridor in the middle is a signboard. What we are doing is a team that can communicate quickly between the two tables.
If you sit on both sides of the table, you can see the face of the other person and you cannot see the monitor, which will form an artificial obstacle, but the artificial obstacle is also psychologically obstacle, and you think it is not your team.
So we turned the vacant space in the middle into a public area, and we were faced with a signboard, and we communicated in time if there was a problem, without looking for a meeting room.
We want to be a team that is easy to communicate with each other. This team has 3 people in the smallest microservice team, 20 people in the largest microservice team, and one of the 20 people is a person who does nothing. He is doing UX page design. Because in our application, our UX is shared by the entire microservice team.
Of course, you can have a "two cars" team or "a big round table team: everyone in the team can just sit down in two cars or a round table in a box when going out for dinner. The main purpose is to reduce team communication and decision-making. cost.
We have three principles for determining the size of the microservice team:
-
The members of the team can communicate with each other at any time: the empty space between the two tables is our meeting room, and we can communicate at any time when we need to, without being disturbed by the table next door.
-
No additional management costs: There is no need to increase the management team to manage the microservice team, and the work responsibility boundary of the microservice team is completely autonomous.
- Do not need to work overtime to complete the planned tasks: show that the current workload is appropriate for the team members.
If it is larger than this size, it proves that your microservice team is too large and needs to be split further. Correspondingly, your microservice development and maintenance workload is too large, and it needs to be further split. The best size of the team is consistent with the workload of the microservice.
If it is smaller than this size, it will increase the management cost because the microservice split is too small. You will find that there are many teams that need to be coordinated, and coordinators have to be added to coordinate the work between the microservices. This is the additional microservice team management cost.
From the perspective of workload, the daily workload must reach a time utilization rate of more than 75%. In other words, if it is the "Nine-to-Six" (9:00-18:00) working method, one hour of lunch break is excluded. There are 8 hours of working time throughout the day, and at least 6 hours must be spent on microservices.
Can have about 2 hours to deal with private and organizational affairs. If the internal working time of the microservice team is less than this ratio, then it proves that there are additional communication costs between organizations. These communications are dependencies that need to be separated or delegated responsibilities.
Organization of the microservice team
What is it like to organize as a microservice team? Our microservices team is a fully functional agile DevOps team. In addition to meeting the above team size, such a team also needs to meet the two conditions of "full function" and "DevOps".
First of all, we are a full-featured team, which means that our team can handle all end-to-end tasks for the entire team.
Secondly, we are a DevOps team, which means that our team has the ability to deploy and release, without the support of the platform group or the operation and maintenance team.
Then, in addition to the general agile team, we also have a head of microservices. Such a team is also called project manager (PM), also called MS-MASTER, it is a composite role, not only the manager or the architect is a technical role. Help us isolate external interference, such as meetings, communication, etc.... to ensure that the team can work independently.
The most important thing is that we need to have a business analyst, our name is BA. The process of discovering the need for such a role is a painful lesson: if no one in your team is very familiar with the business process and organizational structure, the divided microservices will be duplicated with other teams.
This violates the principle of a single source of truth, and early analysis can avoid this. Most of the team is development engineers, ranging from two to 14. Then there is QA. Our QA is not only for testing, our QA also needs to understand the business, and QA is for quality assurance of the entire process.
Our testing process is automated testing written by development engineers.
Note that due to the low cost of abandonment of microservices, we do not have a high unit test coverage, but we have to ensure that the corresponding end-to-end tests and integration tests are mostly covered.
In addition, we need operation and maintenance engineers to help us design continuous deployment and online microservice monitoring.
Establish a delivery specification for microservices
It is very important to establish the specification of microservices. This is also the cornerstone of the successful expansion of microservices.
When we first started doing microservices, we didn't know which microservice path was better. Although there are various methodologies, there will be problems of this kind in the actual process.
So we first do one or two pilots, and then sum up a model, but we must be careful not to form a common code base, otherwise it will become a new aggregation.
Therefore, certain quality specifications must be implemented in each code base to ensure that the quality of each microservice online is guaranteed.
Another point is to create a delivery process that will review the code to the final production environment. To form a specification for the delivery process, this is best automated, and everyone on the team should abide by this specification. We have turned the delivery specification into a series of automated means through the "pipeline as code" technology.
There are also team autonomy activities. At the beginning, when we were doing microservices, it was easy to find North, because you didn't know which ones should be independent.
Therefore, our team decided to explain in one sentence what is the microservice you are doing now? If you don't explain the microservice clearly in one sentence, then your microservice is already too complicated.
In this way, everyone learns through discussions. We use autonomous activities to help everyone improve their understanding of microservices and make some team improvements in the process of doing microservices.
Work rhythm of the microservice team
You can see that the green one in the picture is the MS-MASTER I said. This structure is between each of our product owners and the MS-MASTER of the following microservices, but each structure is a self-organizing structure, and then we Everyone's responsibilities will be changed every once in a while, so that you can think about problems from the perspective of others.
For example, I did development before, I do testing now, I did testing before, I did a piece of DevOps automation work. This ensures that everyone on the team understands the opinions of others in doing this.
Of course, at the beginning, the pace of implementation was very slow. But this learning process can help everyone in the team understand all the work, there is no form of opaque knowledge.
An effective method is pair programming. When you encounter project deadlines and budget deadlines and no more people can be added, the first agile practice that can be cut off is pair programming.
But take a long-term view and you will find that pair programming uses the short-board principle. In your team, you work face-to-face with others. If you work alone, it is difficult for you to feel that your work ability is insufficient. When pairing with others, you will have a clear understanding of your insufficient work ability.
This allows team members to learn from high-level members and continuously improve the team’s shortcomings.
Teams that do not use pair programming are generally the last elimination system. This is easy to understand, but in fact this kind of system will only waste everyone's time.
Because the Matthew effect is followed in such an organization, the strong the stronger and the weak the weaker, which is not conducive to the development of the team. Especially teams that need to replicate.
Another point that needs to be explained is that pair programming is very tiring, because when you are writing code, there is always someone staring at you. You don’t have time to make a difference. You are thinking and designing all the time. If you don’t do this, it’s right. Pairing partners are disrespectful, so from another perspective, pair programming is actually a self-supervision system, allowing the other party to supervise to overcome their own inertia. This is how the high efficiency of our work is formed.
In addition, there is a switch pair between the microservice teams. We often have a problem, that is, some people in our team are high-level, and some people are low-level. High-level people develop some functions and then leave, leaving a pit.
But this risk is often overlooked by us. When we do the handover, we often hand it over without picking it up, and then it becomes another pit. Pair programming is an effective way to avoid pitfalls, but you must pay attention to regularly exchanging paired partners. why?
A very important principle in continuous delivery is to do things that make you painful as soon as possible and frequently. If the resignation of a team member is painful for you and the handover is painful, do the handover once a day.
This is what pair programming and exchanging pair partners do, because every time you do a handover, you practice this. So its handover pressure is very small, but if you leave the system for half a year, the system pressure for you to handover for half a year is very high.
So the obvious disadvantage of pair programming is that it costs people and money. Therefore, pair programming is also a Design for Failure design.
In addition to team communication, we also have regular retrospective meetings. Retrospective meetings are very important, and the team must learn to improve itself. Especially when the micro-service team is just formed, it is necessary to develop the habit of re-listing, otherwise many problems will accumulate and drag down the pace of work of the entire micro-service team.
In addition, you need to give the team full authorization. If you don't trust your team, your team will treat you in a way that does not trust. In this way, everyone will create additional systems to deal with each other instead of cooperating with each other.
Evolving organization
In Netflix's microservice experience, Netflix engineer R. Meshenberg proposed. The landing of microservices requires cooperation and unity in all aspects.
This means that the organizational structure needs to be changed, and the implementation of these changes is difficult. This is not a problem that technology can solve. Moreover, you need your organization to be able to continuously evolve and be able to face various challenges. The changes in organizational structure are often overlooked.
Successful microservice organizations can evolve themselves, and it will bring about technological development. In the next article, we will talk about the characteristics of successful microservice landing technology.