content:
1. What is a distributed system?
2. Why go with a distributed system architecture?
3. How to split the system?
Fourth, the technical challenges brought by distribution?
1. What is a distributed system?
Before talking about the distributed system architecture, let's take a look, what is a distributed system?
Suppose we have a system with more than 300,000 lines of code. Now it is split into 20 small systems, each with more than 10,000 lines of code.
The original code is directly based on the Spring framework to make JVM memory calls. Now it is disassembled, 20 small systems are deployed on different machines, and then an RPC call is made based on a distributed service framework (such as dubbo), interface and interface. Requests and responses are made through network communication.
Therefore, a very important feature of distributed systems is that services need to be called across the network. Let's look at the following figure:
Furthermore, distributed systems can be roughly divided into two categories.
(1) The underlying distributed system
For example, hadoop hdfs (distributed storage system), spark (distributed computing system), storm (distributed streaming computing system), elasticsearch (distributed search system), kafka (distributed publish-subscribe messaging system), etc.
(2) Distributed business system
The distributed business system divides a large system originally developed in java into multiple subsystems, and the multiple subsystems call each other to form a large system as a whole.
For example, suppose you have built an OA system, which includes permission modules, employee modules, leave modules, and financial modules. A project contains a bunch of modules, and modules will call each other. One Machine deployment.
Now if you disassemble his system, the authority system, the employee system, the leave system, the financial system, 4 systems, and 4 projects are deployed on 4 machines.
Then a request comes, and the request is completed. The employee system calls the permission system, calls the leave system, and calls the financial system. The four systems have completed part of the work.
After the last 4 systems are finished, it is considered that the request has been completed. This is called a distributed business system.
Similarly, let's take a picture and feel the above process:
2. Why go with a distributed system architecture?
Some students may want to ask, I have a server running well, all systems and one project are all done, how wonderful. Why do we have to develop a distributed system architecture and call each other remotely, which seems to increase a lot of workload?
Here I will take the blood and tears experience of a company I once worked for as an example to talk about this issue.
Many years ago, before the distributed architecture, the various business lines of the company I worked for were vertical "chimney" projects.
With the rapid development of the Internet, the company's business is also constantly developing, with the increase of registered users, the continuous expansion of the functions and scale of website applications, especially the development of mobile Internet, and the increase of access channels such as APP, WeChat, and self-service terminals. A new kind of business, new demands are constantly pouring in, and the system has encountered various problems.
The first is that the project project becomes uncontrolled and bloated, the system complexity increases, hundreds of thousands of lines of code, dozens of developers, service layer, and dao layer code are copied and used in large quantities, and various code merge conflicts are often dealt with. , very time consuming.
Often I change my code, and someone else calls my interface, which causes problems with his code and needs to be re-tested, which is troublesome.
Then each release is a system with hundreds of thousands of lines of code released together. Everyone has to be anxious to prepare for the launch. When hundreds of thousands of lines of code are launched, every time they go online, they need to do a lot of checks and deal with many abnormal problems. High tension, was made to almost collapse.
And if I have a new business now, I plan to upgrade the relevant dependencies, such as upgrading to the latest spring version, it is not enough, because this may cause other people's code to report errors, and I dare not arbitrarily change the technology. In addition, it takes several minutes for a web project to start each time, and it is very painful to debug the code in the local IDE.
Secondly, with the increase of user access traffic, the system load pressure increases and becomes overwhelmed. By increasing the number of instances, the effect of increasing hardware expansion can be minimal, with frequent failures and low efficiency. The quality of the system is becoming more and more difficult to guarantee, and the test cycle is becoming longer and longer, which cannot meet the needs of the company's business development.
The above are some of the "unbearable" past events of the company I have worked for before. In general, the problems are mainly reflected in the following aspects:
(1) The application code is seriously coupled, and the function expansion is difficult
(2) The development and interaction cycle of new requirements is long, and the testing workload is large
(3) It takes a long time for new development colleagues to become familiar with the system
(4) It is also difficult to upgrade and maintain (the entire system must be upgraded if any changes are made)
(5) It is difficult to improve system performance, with low availability and instability.
Well, now that we have deeply experienced the pain of system coupling, let's take a look at the benefits of system splitting:
First of all, after the system is split, the whole world will feel refreshed.
A system with hundreds of thousands of lines of code is assumed to be divided into 20 services, with an average of 10,000 to 30,000 lines of code per service, and each service is deployed on a separate machine. 20 projects, just use 20 git repository codes, 20 developers, and everyone can maintain their own service.
因为是自己独立的代码，跟别人没关系。再也没有代码冲突了，爽！
每次就测试我自己的代码就可以了，爽！
每次就发布我自己的一个小服务就可以了，爽！
技术上想怎么升级就怎么升级，保持接口定义不变，输入输出内容不变就可以了，爽！To sum up, after the distributed system is split, the development efficiency of large teams of complex systems can be greatly improved.
3. How to split the system?
Generally speaking, to split the system, you first need to be familiar with the system as a whole. You can take multiple rounds of splitting ideas. The first splitting is to split the previous large modules in a coarse-grained manner.
For example, an e-commerce system can be divided into order system, commodity system, store system, membership system, promotion system, payment system and so on.
Later, each system may become more and more complicated. For example, the order system can be further split into shopping cart system, inventory system, price system, etc.
In general, it is based on the idea of domain-driven design and the summary of actual combat experience. At the same time, referring to some common practices in the industry, everyone discusses and splits, gradually optimizes, splits in multiple rounds, runs in small steps, and finally achieves a better state. .
Fourth, the technical challenges brought by distribution?
The first is the selection of the distributed service framework. At present, the mainstream ones in China are still dubbo and spring cloud.
Let's think about it, what problem is the service framework mainly used to solve? Is it possible to do distributed architecture without dubbo or spring cloud?
It is of course possible to not use service frameworks such as dubbo or spring cloud, but this requires you to handle a lot of things yourself.
For example, if each subsystem uses the restful interface to call, then it is an http call. At this time, for example, if an object is sent in the past, it is necessary to make a JSON by itself, and then retry after a call fails?
In addition, generally speaking, it is a cluster deployment, and the target system has multiple instances, so I have to write a load balancing algorithm. How to randomly select one of the multiple target machines to call each time?
In addition, if the target system expands and deploys a new instance, or if the server fails and an instance goes offline, how can the caller be aware of it dynamically? There are many problems like this. If you don't use the service framework, you will encounter all kinds of problems if you mess around like this.
The above process is presented to you with a picture:
If you choose a distributed service framework, you need to deeply grasp the use and underlying principles of this framework. For example, dubbo needs to understand the following issues:
(1) How does dubbo work?
(2) What serialization protocols are supported by dubbo?
(3) What is the load balancing and high availability strategy of dubbo? Dynamic proxy strategy?
(4), Dubbo's SPI idea?
(5) How to perform service governance, service degradation, failure retry and timeout retry based on dubbo?
(6) How to design the idempotency of the dubbo service interface (for example, no repeated deduction, no repeated order generation, no repeated card number creation)?
(7) How to ensure the order of the dubbo service interface requests?
(8) How to design an rpc framework similar to dubbo?
The same is true for using spring cloud, such as how does eureka work? The principle of feign declarative call? Wait for the underlying principles to be understood.
There are other common technical problems to be solved after taking a distributed architecture:
(1) Distributed session
(2) Distributed lock
(3) Distributed transactions
(4) Distributed search
(5) Distributed cache
(6) Distributed message queue
(7) Unified configuration center
(8) Distributed storage, database sub-database sub-table
(9), current limiting, fusing, downgrading, etc.
The above problems, in depth, each point needs to be elaborated in N articles. It is impossible to expand one by one here. We will continue to talk about various technical issues under these distributed architectures through some articles later.
Reprinted in: https://blog.51cto.com/14257001/2408391