Taobao's 10-year Architecture Evolution

stand-alone architecture
The first evolution: Tomcat is deployed separately from the database
The second evolution: the introduction of local cache and distributed cache
The third evolution: the introduction of reverse proxy to achieve load balancing
The fourth evolution: database read and write separation
Fifth Evolution: Database Segmentation by Business
The sixth evolution: split the large table into small tables
The seventh evolution: use LVS or F5 to balance multiple Nginx loads
Eighth Evolution: Implementing Server Room Load Balancing Through DNS Round Robin
Ninth Evolution: Introducing technologies such as NoSQL databases and search engines
The tenth evolution: large applications are split into small applications
The Eleventh Evolution: Reusable Functions Separated into Microservices
Twelfth Evolution: Introducing the Enterprise Service Bus (ESB) to shield access differences of service interfaces
The Thirteenth Evolution: Introducing Containerization Technology to Realize Operating Environment Isolation and Dynamic Service Management
The Fourteenth Evolution: Carrying the System on the Cloud Platform

1 Overview

This article takes Taobao as an example to introduce the evolution process of the server-side architecture from one hundred to tens of millions of concurrent cases. At the same time, it lists the related technologies that will be encountered in each evolution stage, so that everyone can have an overall understanding of the evolution of the architecture. As we know, the article concludes with a summary of some architectural design principles.

Special Note: This article uses Taobao as an example just to illustrate the problems that may be encountered in the evolution process, not the real technological evolution path of Taobao

2. Basic concepts

Before introducing the architecture, in order to prevent some readers from being ignorant of some concepts in architecture design, the following introduces the most basic concepts:

Multiple modules in a distributed system are deployed on different servers, which can be called a distributed system. For example, Tomcat and the database are deployed on different servers, or two Tomcats with the same function are deployed on different servers .
When some nodes in a high-availability system fail, other nodes can take over and continue to provide services, then the system can be considered to have high availability
Cluster A domain-specific software is deployed on multiple servers and provides a class of services as a whole, which is called a cluster. For example, the Master and Slave in Zookeeper are deployed on multiple servers respectively, forming a whole to provide centralized configuration services. In a common cluster, the client can often connect to any node to obtain services, and when a node in the cluster goes offline, other nodes can often automatically take over and continue to provide services, which means that the cluster has high availability
When a load balancing request is sent to the system, the request is evenly distributed to multiple nodes in some way, so that each node in the system can evenly process the request load, then the system can be considered to be load balanced
When the forward proxy and reverse proxy systems want to access the external network, the request is forwarded through a proxy server. From the perspective of the external network, it is the access initiated by the proxy server. At this time, the proxy server implements the forward proxy; when the external request When entering the system, the proxy server forwards the request to a certain server in the system. For external requests, only the proxy server interacts with it. At this time, the proxy server implements a reverse proxy. To put it simply, forward proxy is a process in which a proxy server replaces the internal system to access the external network, and reverse proxy is a process in which external requests to access the system are forwarded to the internal server through the proxy server.

3. Architecture Evolution

3.1 Stand-alone architecture

Take Taobao as an example. At the beginning of the website, the number of applications and users were small, so Tomcat and the database could be deployed on the same server. When the browser initiates a request to www.taobao.com, it first converts the domain name to the actual IP address 10.102.4.1 through the DNS server (Domain Name System), and then the browser accesses Tomcat corresponding to this IP.

As the number of users increases, Tomcat and the database compete for resources, and the performance of a single machine is not enough to support the business

3.2 The first evolution: Tomcat is deployed separately from the database

Tomcat and the database monopolize server resources respectively, significantly improving their respective performance.

As the number of users grows, concurrent reads and writes to the database become a bottleneck

3.3 The second evolution: the introduction of local cache and distributed cache

Add a local cache on the same Tomcat server or in the same JVM, and add a distributed cache externally to cache popular product information or html pages of popular products, etc. Through caching, most requests can be intercepted before reading and writing to the database, greatly reducing the pressure on the database. The technologies involved include: using memcached as a local cache, using Redis as a distributed cache, and also involves issues such as cache consistency, cache penetration/breakdown, cache avalanche, and invalidation of hot data sets.

The cache resists most of the access requests. As the number of users increases, the concurrency pressure mainly falls on the stand-alone Tomcat, and the response gradually slows down.

3.4 The third evolution: the introduction of reverse proxy to achieve load balancing

Deploy Tomcat on multiple servers, and use reverse proxy software (Nginx) to evenly distribute requests to each Tomcat. It is assumed here that Tomcat supports up to 100 concurrency, and Nginx supports up to 50,000 concurrency. In theory, Nginx can resist 50,000 concurrency by distributing requests to 500 Tomcats. The technologies involved include: Nginx and HAProxy, both of which are reverse proxy software working on the seventh layer of the network. They mainly support the http protocol, and also involve session sharing, file upload and download.

The reverse proxy greatly increases the concurrency that the application server can support, but the increase in concurrency also means that more requests penetrate to the database, and the stand-alone database eventually becomes the bottleneck

3.5 The fourth evolution: database read-write separation

Divide the database into read and write databases. There can be multiple read databases. The data in the write database is synchronized to the read database through the synchronization mechanism. For the scene where you need to query the latest written data, you can write an extra copy in the cache. The cache gets the latest data. The technologies involved include: Mycat, which is a database middleware, through which the separate reading and writing of the database and sub-database sub-tables can be organized. The client can access the underlying database through it, and it will also involve data synchronization and data consistency issues. .

There are more and more businesses, and there is a large gap in the number of visits between different businesses. Different businesses directly compete with the database and affect each other's performance.

3.6 The Fifth Evolution: Database Segmentation by Business

Save the data of different businesses in different databases to reduce resource competition between businesses. For businesses with a large number of visits, more servers can be deployed to support them. This also makes it impossible to directly perform correlation analysis on cross-business tables, and needs to be solved through other means, but this is not the focus of this article. Those who are interested can search for solutions by themselves.

With the increase of the number of users, the single-machine write library will gradually reach the performance bottleneck

3.7 The sixth evolution: splitting large tables into small tables

For example, for comment data, it can be hashed according to the product ID and routed to the corresponding table for storage; for payment records, a table can be created by hour, and each hour table can be split into small tables, and user ID or record number can be used to route data . As long as the amount of table data for real-time operations is small enough and requests can be evenly distributed to small tables on multiple servers, the database can improve performance through horizontal expansion. The aforementioned Mycat also supports access control when large tables are split into small tables.

This approach significantly increases the difficulty of database operation and maintenance, and has higher requirements for DBAs. When the database is designed with this structure, it can already be called a distributed database, but this is only a logical database as a whole. Different components in the database are implemented by different components, such as the management and request of sub-database and sub-table. Distribution is implemented by Mycat, SQL parsing is implemented by a stand-alone database, read-write separation may be implemented by a gateway and a message queue, and the summary of query results may be implemented by a database interface layer, etc. This architecture is actually an MPP (large-scale A class of implementations of the Parallel Processing) architecture.

At present, there are many MPP databases in both open source and commercial use. Greenplum, TiDB, Postgresql XC, HAWQ, etc. are more popular in open source, and commercial ones include GBase of NTU General, Snowball DB of Ruifan Technology, LibrA of Huawei, etc. Different MPP databases have different emphases. For example, TiDB focuses more on distributed OLTP scenarios, and Greenplum focuses more on distributed OLAP scenarios. These MPP databases basically provide SQL standard support capabilities similar to Postgresql, Oracle, and MySQL. It can parse a query into a distributed execution plan and distribute it to each machine for parallel execution. Finally, the database itself summarizes the data and returns it. It also provides capabilities such as authority management, sub-database sub-table, transaction, and data copy. It can support clusters of more than 100 nodes, greatly reducing the cost of database operation and maintenance, and enabling the database to achieve horizontal expansion.

Both the database and Tomcat can be expanded horizontally, and the supported concurrency is greatly increased. As the number of users grows, the stand-alone Nginx will eventually become the bottleneck

3.8 The seventh evolution: use LVS or F5 to balance multiple Nginx loads

Since the bottleneck is Nginx, it is impossible to achieve load balancing of multiple Nginxes through two layers of Nginx. LVS and F5 in the figure are load balancing solutions working on the fourth layer of the network. LVS is software that runs in the kernel state of the operating system and can forward TCP requests or higher-level network protocols. Therefore, the supported protocols are more It is rich, and its performance is much higher than that of Nginx. It can be assumed that a single-machine LVS can support hundreds of thousands of concurrent request forwarding; F5 is a kind of load balancing hardware, which is similar to the capabilities provided by LVS and has higher performance than LVS, but it is expensive .

Since LVS is a stand-alone software, if the server where LVS is located goes down, the entire back-end system will be inaccessible, so a backup node is required. You can use the keepalived software to simulate a virtual IP, and then bind the virtual IP to multiple LVS servers. When the browser accesses the virtual IP, it will be redirected to the real LVS server by the router. When the main LVS server is down, the keepalived software will Automatically update the routing table in the router, and redirect the virtual IP to another normal LVS server, so as to achieve the effect of high availability of the LVS server.

It should be noted here that the drawing from the Nginx layer to the Tomcat layer in the above figure does not mean that all Nginx forward requests to all Tomcats. In actual use, it may be that several Nginxes are connected to a part of Tomcat. High availability is achieved through keepalived, and other Nginx is connected to another Tomcat, so that the number of Tomcats that can be accessed can be doubled.

Because LVS is also stand-alone, as the number of concurrency increases to hundreds of thousands, the LVS server will eventually reach the bottleneck. At this time, the number of users reaches tens of millions or even hundreds of millions. Users are distributed in different regions and the distance from the server room is different, resulting in The latency of the access will be significantly different

3.9 The Eighth Evolution: Implementing Server Room Load Balancing Through DNS Polling

A domain name can be configured in the DNS server to correspond to multiple IP addresses, and each IP address corresponds to a virtual IP in a different computer room. When a user visits www.taobao.com, the DNS server will use a round-robin strategy or other strategies to select an IP for the user to access. This method can realize the load balance of the computer room. So far, the system can achieve the horizontal expansion of the computer room level. The concurrent volume of tens of millions to hundreds of millions can be solved by adding the computer room. The concurrent request at the entrance of the system is no longer a problem. .

With the richness of data and the development of business, the requirements for retrieval and analysis are becoming more and more abundant. Relying on the database alone cannot solve such a rich demand.

3.10 The Ninth Evolution: Introducing NoSQL Database and Search Engine Technologies

When the amount of data in the database reaches a certain scale, the database is not suitable for complex queries, and often only meets common query scenarios. For statistical report scenarios, the results may not be able to be run when the amount of data is large, and other queries will slow down when running complex queries. For scenarios such as full-text retrieval and variable data structures, databases are inherently unsuitable. Therefore, it is necessary to introduce a suitable solution for a specific scenario.

For example, for massive file storage, it can be solved by distributed file system HDFS; for key value type data, it can be solved by solutions such as HBase and Redis; for full-text retrieval scenarios, it can be solved by search engines such as ElasticSearch; for multi-dimensional analysis scenarios, it can be solved Solve through solutions such as Kylin or Druid.

Of course, the introduction of more components will also increase the complexity of the system. The data saved by different components needs to be synchronized, the issue of consistency needs to be considered, and more operation and maintenance methods are needed to manage these components.

The introduction of more components solves the rich requirements, and the business dimension can be greatly expanded, followed by an application that contains too many business codes, making it difficult to upgrade and iterate the business

3.11 The Tenth Evolution: Splitting Large Applications into Small Applications

Divide the application code according to the business sector, so that the responsibilities of a single application are clearer, and they can be upgraded and iterated independently of each other. At this time, some common configurations may be involved between applications, which can be resolved through the distributed configuration center Zookeeper.

There are shared modules between different applications, and separate management by the application will result in multiple copies of the same code, resulting in the upgrade of all application codes when public functions are upgraded

3.12 The Eleventh Evolution: Multiplexed Functions Separated into Microservices

If user management, order, payment, authentication and other functions exist in multiple applications, then the codes of these functions can be extracted separately to form a separate service for management. Such services are so-called microservices, applications and services Public services are accessed through multiple methods such as HTTP, TCP or RPC requests, and each individual service can be managed by a separate team. In addition, Dubbo, SpringCloud and other frameworks can be used to implement functions such as service governance, current limiting, circuit breaking, and downgrade to improve service stability and availability.

The interface access methods of different services are different, and the application code needs to be adapted to multiple access methods to use the service. In addition, when the application accesses the service, the services may also access each other, and the call chain will become very complicated and the logic will become confusing.

3.13 The Twelfth Evolution: Introducing the Enterprise Service Bus (ESB) to Shield Service Interface Access Differences

The access protocol conversion is performed uniformly through the ESB, and the application uniformly accesses the back-end services through the ESB, and the services also call each other through the ESB, so as to reduce the coupling degree of the system. This single application is split into multiple applications, public services are extracted separately for management, and the enterprise message bus is used to decouple the coupling between services. This is the so-called SOA (service-oriented) architecture, which is similar to microservices. Architectures are confusing because the representations are very similar. Personal understanding, microservice architecture refers more to the idea of extracting public services in the system for separate operation and maintenance management, while SOA architecture refers to an architectural idea that splits services and makes service interface access unified. SOA architecture Contains the idea of microservices.

With the continuous development of business, there will be more and more applications and services, and the deployment of applications and services will become complicated. Deploying multiple services on the same server also needs to solve the problem of operating environment conflicts. In addition, dynamic scaling is required for such things as big promotions. In the scenario where the service capacity needs to be expanded horizontally, it is necessary to prepare the operating environment for the newly added service, deploy the service, etc., and the operation and maintenance will become very difficult.

3.14 The Thirteenth Evolution: Introducing Containerization Technology to Realize Operating Environment Isolation and Dynamic Service Management

At present, the most popular containerization technology is Docker, and the most popular container management service is Kubernetes (K8S). Applications/services can be packaged as Docker images, and images can be dynamically distributed and deployed through K8S. The Docker image can be understood as a minimal operating system that can run your application/service, which contains the running code of the application/service, and the operating environment is set according to actual needs. After the entire "operating system" is packaged into a mirror, it can be distributed to machines that need to deploy related services, and the service can be started by directly starting the Docker mirror, making the deployment and operation and maintenance of the service easier.

Before the big promotion, you can divide the server on the existing machine cluster to start the Docker image to enhance the performance of the service. After the big promotion, you can turn off the mirror without affecting other services on the machine (before section 3.14, It is necessary to modify the system configuration to adapt the service to run the service on the newly added machine, which will destroy the operating environment required by other services on the machine).

After using containerization technology, the problem of dynamic expansion and contraction of services can be solved, but the machine still needs to be managed by the company itself. When it is not a big promotion, it still needs to idle a lot of machine resources to deal with the big promotion, the cost of the machine itself and the operation and maintenance cost Both are extremely high, with low resource utilization

3.15 The Fourteenth Evolution: Carrying the System on the Cloud Platform

The system can be deployed on the public cloud, using the massive machine resources of the public cloud to solve the problem of dynamic hardware resources. During the period of big promotion, temporarily apply for more resources in the cloud platform, and combine Docker and K8S to quickly deploy services , Release resources after the big promotion ends, truly pay on demand, greatly improve resource utilization, and greatly reduce operation and maintenance costs.

The so-called cloud platform is to abstract massive machine resources into a whole resource through unified resource management, on which hardware resources (such as CPU, memory, network, etc.) can be dynamically applied for on demand, and general operations are provided on it The system provides commonly used technical components (such as Hadoop technology stack, MPP database, etc.) , mail service, personal blog, etc.). The following concepts are involved in the cloud platform:

IaaS: Infrastructure as a Service. Corresponding to the unification of machine resources mentioned above as a whole of resources, the level of hardware resources can be dynamically applied;
PaaS: Platform as a Service. Corresponding to the above-mentioned provision of commonly used technical components to facilitate the development and maintenance of the system;
SaaS: software as a service. Corresponding to the provision of developed applications or services mentioned above, payment is made according to functional or performance requirements.

So far, the above-mentioned problems from high concurrent access to service architecture and system implementation have their own solutions, but at the same time, it should be realized that in the above introduction, such as cross Practical issues such as data synchronization in the computer room, implementation of distributed transactions, etc., these issues will be discussed separately in the future

4. Summary of architecture design

Does the adjustment of the structure have to be carried out according to the above-mentioned evolution path? No, the sequence of architectural evolution mentioned above is just a single improvement for a certain aspect. In actual scenarios, there may be several problems that need to be solved at the same time, or it may be another aspect that reaches the bottleneck first. At this time, It should be solved according to actual problems. For example, in the scenario where the amount of concurrency in the government class may not be large, but the business may be rich, high concurrency is not the key problem to be solved. At this time, the priority may be a solution with rich needs.
To what extent should the architecture be designed for the system to be implemented? For a system that is implemented once and has clear performance indicators, it is sufficient to design the architecture to support the performance requirements of the system, but an interface to extend the architecture should be left in case it is needed. For a constantly evolving system, such as an e-commerce platform, it should be designed to meet the requirements of the next stage of user volume and performance indicators, and iteratively upgrade the architecture according to business growth to support higher concurrency and richer business .
What is the difference between server-side architecture and big data architecture? The so-called "big data" is actually a general term for scenario solutions such as massive data collection, cleaning and conversion, data storage, data analysis, and data services. Each scenario includes a variety of optional technologies, such as data collection with Flume, Sqoop, Kettle, etc., data storage includes distributed file system HDFS, FastDFS, NoSQL database HBase, MongoDB, etc., data analysis includes Spark technology stack, machine learning algorithms, etc. In general, big data architecture is an architecture that integrates various big data components according to business needs, and generally provides capabilities such as distributed storage, distributed computing, multi-dimensional analysis, data warehouse, and machine learning algorithms. The server-side architecture refers more to the architecture at the application organization level, and the underlying capabilities are often provided by the big data architecture.
Are there some principles of architectural design?
- N+1 design. Every component in the system should have no single point of failure;
- Rollback design. Ensure that the system is forward compatible, and there should be a way to roll back the version when the system is upgraded;
- Disable design. It should provide a configuration to control the availability of specific functions, and the function can be quickly offline when the system fails;
- Monitor design. The means of monitoring should be considered at the design stage;
- Multi-active data center design. If the system requires extremely high availability, it should be considered to implement multi-active data centers in multiple places, so that the system is still available when at least one computer room is powered off;
- Adopt mature technology. Newly developed or open source technologies often have many hidden bugs, and if something goes wrong without commercial support, it may be a disaster;
- Resource isolation design. A single business should avoid occupying all resources;
- The architecture should be able to scale horizontally. Only when the system can be expanded horizontally can the bottleneck problem be effectively avoided;
- Buy non-core. If non-core functions require a lot of R&D resources to solve, consider purchasing mature products;
- Use commodity hardware. Commercial hardware can effectively reduce the probability of hardware failure;
- Iterate quickly. The system should quickly develop small functional modules, go online for verification as soon as possible, and find problems early to greatly reduce the risk of system delivery;
- Stateless design. The service interface should be made stateless, and the access to the current interface does not depend on the state of the last access to the interface.