Internet Architecture [High Concurrency]

What is high concurrency?

High concurrency is one of the factors that must be considered in the design of the Internet distributed system architecture. It usually means that the design ensures that the system can process many requests in parallel at the same time.

Some commonly used indicators related to high concurrency are: Response Time (Response Time), Throughput (Throughput), Query Per Second (Query Per Second), and the number of concurrent users.

Response time: The time it takes for the system to respond to the request. For example, it takes 200ms for the system to process an HTTP request, and this 200ms is the response time of the system.

Throughput: The number of requests processed per unit time.

QPS: The number of response requests per second. In the Internet domain, the distinction between this metric and throughput is not so clear.

Number of concurrent users: The number of users who are using the system functions normally at the same time. For example, in an instant messaging system, the number of simultaneous online users represents the number of concurrent users of the system to a certain extent.

How to improve the concurrency capability of the system The

Internet distributed architecture design, and the way to improve the system concurrency capability, there are two main methodologies: vertical scaling (Scale Up) and horizontal scaling (Scale Out).

Vertical expansion: Improve the processing capacity of a single machine. There are two ways of vertical expansion:

   (1) Enhance the hardware performance of a single machine, for example: increase the number of CPU cores such as 32 cores, upgrade a better network card such as 10 Gigabit, upgrade a better hard disk such as SSD, and expand the hard disk capacity such as 2T, Expand the system memory such as 128G;
   (2) Improve the performance of the stand-alone architecture, for example: use Cache to reduce the number of IOs, use asynchrony to increase the throughput of a single service, and use a lock-free data structure to reduce the response time;


in the early days of the rapid development of Internet business , if budget is not a problem, it is strongly recommended to use "enhancing single-machine hardware performance" to improve the system's concurrency capability, because at this stage, the company's strategy is often to seize time for business development, and "enhancing single-machine hardware performance" is often the fastest method.

Whether it is to improve the performance of stand-alone hardware or to improve the performance of stand-alone architecture, there is a fatal deficiency: stand-alone performance is always limited. Therefore, the ultimate solution for the design of high concurrency in the distributed architecture of the Internet is horizontal expansion.


Horizontal expansion: As long as the number of servers is increased, the system performance can be linearly expanded. Horizontal expansion requires system architecture design. How to carry out horizontally scalable design at each layer of the architecture, as well as the common horizontal expansion practices at each layer of the Internet company's architecture, are the focus of this article.


Common Internet Layered Architecture

Common Internet distributed architecture is as above, divided into:
  (1) Client layer: the typical caller is a browser browser or mobile application APP
  (2) Reverse proxy layer: system entry, reverse proxy
  (3 ) Site application layer: implement core application logic and return html or json
  (4) service layer: if service is realized, there is this layer
  (5) data-cache layer: cache accelerates access to storage
  (6) data-database layer : Database solidified data storage How are the horizontal expansion of

the entire system at all levels implemented?


Hierarchical horizontal expansion architecture practice

[Horizontal expansion of the

reverse proxy layer] The horizontal expansion of the reverse proxy layer is achieved through "DNS polling": dns-server configures multiple resolution IPs for a domain name, and each DNS resolution Requests to access dns-server will poll these ips.

When nginx becomes the bottleneck, as long as the number of servers is increased, the deployment of the nginx service is added, and an external network IP is added, the performance of the reverse proxy layer can be expanded, and the theoretically infinitely high concurrency can be achieved.

[Horizontal expansion of the

site layer] The horizontal expansion of the site layer is realized through "nginx". By modifying nginx.conf, multiple web backends can be set up.
When the web backend becomes a bottleneck, as long as the number of servers is increased, the deployment of new web services is added, and a new web backend is configured in the nginx configuration, the performance of the site layer can be expanded, achieving theoretically infinitely high concurrency.

[Horizontal expansion of the

service layer] The horizontal expansion of the service layer is realized through the "service connection pool".

When the site layer calls the downstream service layer RPC-server through the RPC-client, the connection pool in the RPC-client will establish multiple connections with the downstream service. When the service becomes a bottleneck, just increase the number of servers and deploy new services. Establishing a new downstream service connection at the RPC-client can expand the performance of the service layer and achieve theoretically infinitely high concurrency. If you need to perform automatic expansion of the service layer elegantly, you may need to support the automatic service discovery function in the configuration center.

[Horizontal expansion of the data layer]

In the case of a large amount of data, the data layer (cache, database) involves the horizontal expansion of data, and the data (cache, database) originally stored on one server is horizontally split to different servers up, in order to achieve the purpose of expanding system performance.

There are several common horizontal splitting methods in the Internet data layer. Take the database as an example:

[Horizontal splitting according to scope]

Each data service stores a certain range of data:
user0 library, storage uid range 1-1kw
user1 library, storage

The advantages of this scheme with a uid range of 1kw-2kw are:
(1) The rules are simple, and the service can route to the corresponding storage service only by judging the uid range;
(2) The data balance is better;
(3) It is easier to expand and can Add a uid[2kw,3kw] data service at any time; the

deficiency is:
(1) The load of the request is not necessarily balanced. Generally speaking, the newly registered users will be more active than the old users, and the service request pressure of the large range will be greater;

[Split according to the hash level]

Each database stores a certain Part of the data after the hash of the key value:
user0 library, storing even-numbered uid data
user1 library, storing odd-numbered uid data The advantages of

this scheme are:
(1) The rules are simple, and the service only needs to hash the uid to route to the corresponding storage service;
( 2) The data balance is good;
(3) The request uniformity is good;

the shortcomings are:
(1) It is not easy to expand, expand a data service, when the hash method is changed, data migration may be required;

it should be noted here that through Horizontal splitting to expand system performance is fundamentally different from the way master-slave synchronous read-write separation to expand database performance.

Extend database performance through horizontal splitting:
(1) The amount of data stored on each server is 1/n of the total, so the performance of a single machine will also be improved;
(2) If the data on n servers does not intersect, which server The union of the above data is the complete set of data;
(3) The data is horizontally split to n servers, theoretically, the read performance is expanded by n times, and the write performance is also expanded by n times (in fact, it is far more than n times, because the data of a single server is expanded by n times. (1)

The amount
of data stored on each server is the same as the total amount;
(2) The data on n servers are the same, all (3) Theoretically, the read performance is
expanded by n times, the write is still a single point, and the write performance remains unchanged;

The horizontal splitting of the cache layer is similar to the horizontal splitting of the database layer, and most of them are range splitting and hash splitting, so they will not be expanded.

Summary

High concurrency is one of the factors that must be considered in the design of the Internet distributed system architecture. It usually means that the design ensures that the system can process many requests in parallel at the same time.

There are two main methods to improve the concurrency capability of the system: vertical expansion (Scale Up) and horizontal expansion (Scale Out).
The former vertical expansion can improve concurrency by improving the performance of single-machine hardware or the performance of single-machine architecture, but there is always a limit to single-machine performance. The ultimate solution for high concurrency in Internet distributed architecture design is the latter: horizontal expansion.

In the layered architecture of the Internet, the practice of horizontal expansion at each level is different:
(1) The reverse proxy layer can be extended horizontally through "DNS round-robin";
(2) The site layer can be extended horizontally through nginx ;
(3) The service layer can be extended horizontally through the service connection pool;
(4) The database can be extended horizontally according to the data range or data hash;

after the horizontal expansion of each layer, the number of servers can be increased. To improve the performance of the system, the theoretical performance is infinite.