1. What is a distributed system?
Before talking about the distributed system architecture, let's first understand what a distributed system is. To explain what a distributed system is in vernacular, it is assumed that there was a system with more than 200,000 lines of code, and now it is split into 20 small systems, each with more than 10,000 lines of code. The original code was directly based on spring calls, but now it is disassembled. 20 small systems are deployed on different machines, and an RPC call has to be made based on a distributed service framework (such as dubbo), and the interface is connected through the network. communication to request and respond.
Distributed systems can be roughly divided into two categories.
1. The underlying distributed system.
Such as hadoop hdfs (distributed storage system), hadoop mapreduce, spark (distributed computing system), storm (distributed streaming computing system), elasticsearch (distributed search system), kafka (distributed publish-subscribe messaging system), etc.
2. Distributed business system.
A distributed business system is, for example, splitting a large system originally developed in Java into multiple subsystems, and the multiple subsystems call each other to form a large overall system. For example, an e-commerce system is generally divided into commodity centers, shopping carts, order centers, promotion centers, payment centers, WMS, inventory centers, logistics centers, etc. Each center will be made into an independent subsystem, deployed independently, and each The subsystems call and cooperate with each other to form a large-scale e-commerce system. The previous vernacular to explain what is a distributed system is also a distributed business system.
Second, why do distributed system architecture? (Why system split?)
background:
Before the distributed architecture, the company's various business lines were vertical "chimney-type" projects. With the rapid development of the Internet, the company's business is also constantly developing, with the increase of registered users, the function and scale of website applications. With the continuous expansion, especially the development of the mobile Internet, the increase of access channels such as APP, WeChat, and self-service terminals, various new businesses and new demands are constantly pouring in, and the system has encountered various problems. The first is that the project project becomes uncontrolled and bloated, the system complexity increases, hundreds of thousands of lines of code, dozens of developers, service layer, and dao layer code are copied and used in large quantities, and various code merge conflicts are often dealt with. , very time consuming. Often I change my code, and someone else calls me, causing problems with his code, and needs to be re-tested, which is troublesome. Then every release is a system with hundreds of thousands of lines of code released together. Everyone has to prepare for the launch together. When hundreds of thousands of lines of code are launched, a lot of inspections may be required each time they go online, and many abnormal problems are handled. Both troublesome and painful. And if I have a new business now, I plan to upgrade the relevant dependencies, such as upgrading to the latest spring version, it is not enough, because this may cause other people's code to report errors, and I dare not arbitrarily change the technology. It takes at least 3 minutes for a web project to start each time, and it is very painful to debug the code in the local eclipse. Secondly, with the increase of user access traffic, the system load pressure increases and becomes overwhelmed. By increasing the number of instances, the effect of increasing hardware capacity expansion has been minimal, with frequent failures and low efficiency. The quality of the system is becoming more and more difficult to guarantee, and the test cycle is becoming longer and longer, which cannot meet the needs of the company's business development.
In general, the problems are mainly reflected in the following aspects:
1. Application code coupling is serious, and function expansion is difficult.
2. The development and interaction cycle of new requirements is long, and the testing workload is large.
3. It takes a long time for new development colleagues to become familiar with the system.
4. It is also difficult to upgrade and maintain (to change any point, the entire system must be upgraded).
5. It is difficult to improve system performance, with low availability and instability.
The benefits of system splitting:
After the system is split, the whole world will feel refreshed. A system with hundreds of thousands of lines of code is assumed to be divided into 20 services, with an average of 10,000 to 30,000 lines of code per service, and each service is deployed on a separate machine. For 20 projects, use 20 git repository codes and 20 developers. Everyone can maintain their own service, because it is their own independent code and has nothing to do with others. No more code conflicts, cool. Every time I test my own code, it's cool. Every time I publish a small service of my own, it's cool. Technically, you can upgrade how you want to upgrade, keep the interface definition unchanged, and the input and output content unchanged. It's cool. After the distributed system is split, the development efficiency of large teams of complex systems can be greatly improved.
3. How to split the system?
Generally speaking, to split the system, you first need to be familiar with the system as a whole. You can go for multiple rounds of splits. For the first split, you can just shoot your head based on experience, that is, split the previous large modules. For example, our e-commerce system can be divided into order system, commodity system, store system, membership system, promotion system, payment system and so on. Later, each system may become more and more complicated. For example, the order system can be divided into shopping cart system, inventory system, price system, order management system, etc. In general, it is based on the idea of the domain-driven model and the summary of actual combat experience. Everyone discusses and divides it, gradually, in multiple rounds, and runs in small steps, and finally reaches a better state.
4. How to adopt a distributed architecture? (What problems (technical challenges) will be encountered after taking a distributed architecture?)
The first is the selection of the distributed service framework. At present, the mainstream ones in China are still dubbo and spring cloud. Spring cloud has only become popular in the last year or two. In 2015, spring cloud was immature. Dubbo was widely used in the industry. We naturally chose dubbo.
What problem is the service framework mainly used to solve? (Is it possible to do a distributed architecture without dubbo?) Of course, it is possible to do without service frameworks such as dubbo, but this requires you to handle a lot of things yourself. For example, if each subsystem uses the restful interface call, then it is an http call. At this time, for example, if an object is sent in the past, it must be made into a json by itself, and then retry after a call fails? Generally speaking, it is a cluster deployment, and the target system has multiple instances, so I have to write a load balancing algorithm. How to randomly select one of the multiple target machines to call each time? If an instance is added to the target system and an instance is disabled due to a server failure, how can the caller be aware of it dynamically? Waiting for a lot of problems, if you don't use the service framework, you will encounter all kinds of problems if you mess around like this.
After the selection of the distributed service framework is determined, the second is to learn the main content of the dubbo framework and other common problems to be solved after the distributed architecture.
The main things to learn after using dubbo.
1. How does dubbo work?
2. What serialization protocols are supported by dubbo?
3. Dubbo's load balancing and high availability strategy? Dynamic proxy strategy?
4. Dubbo's SPI idea?
5. How to perform service governance, service degradation, failure retry and timeout retry based on dubbo?
6. How to design the idempotency of the dubbo service interface (for example, no repeated deduction, no repeated order generation, no repeated card number creation)?
7. How to ensure the order of requests from the dubbo service interface?
Some other common problems to be solved after taking a distributed architecture.
1. Distributed session.
2. Distributed lock.
3. Distributed transactions.
4. Unified configuration center.
5. High-performance, high-availability architecture: message queue, cache, current limiting, circuit breaker, downgrade, monitoring, database sub-database sub-table, etc.
For the study of the main content of the dubbo framework and other common problems to be solved after the distributed architecture, I am going to write some articles for each knowledge point to explain.