Article from: Alibaba's billion-level concurrent system design (2021 version)
Link: https://pan.baidu.com/s/1lbqQhDWjdZe1CBU-6U4jhA Extraction code: 8888
table of Contents
Next, let us formally enter the course.
So why does frequent connection creation cause slow response time? Let's look at an actual test.
Use connection pool to establish database connection in advance
Pre-create threads with thread pool
In the previous few courses, I took you from a macro perspective to understand the basics of high-concurrency system design. As you already know, the purpose of our system design is to obtain better performance, higher availability, and stronger System expansion capabilities.
So starting from this lecture, we formally enter the evolution chapter. I will start from the part and take you one by one to understand some of the methods used to complete these goals. These methods will specifically solve the problems in the design of high concurrency systems. For example, in the 15th lecture, I will mention the Bloom filter. This component is to solve the problem of how to improve the cache hit rate as much as possible when there is a large amount of cache penetration.
Of course, simply explaining the theory and explaining the plan will be boring, so I will use a virtual system as the main line throughout the course to explain what problems we will encounter when the system reaches a certain stage, and then what kind of What are the technical points involved in the response process? Through this way of telling, try to use cases to lead to problems,
It can let you know what the solution is when you encounter different problems. Of course, in this process, I hope you can think more, and then apply the knowledge you learn to actual projects.
Next, let us formally enter the course.
Imagine such a scenario. One day, the CEO of the company called you to the conference room and told you that the company saw a new business opportunity. I hope you can lead a brother to quickly develop an e-commerce system for a certain vertical field. . In the case of manpower shortage and insufficient time, in order to be able to complete the task, you did not hesitate to adopt the simplest architecture: a Web server on the front end runs business code, and a database server on the back end stores business data. This architecture diagram is the most familiar and simple architecture prototype for each of us. Many systems look like this at the beginning, but as the business complexity increases, the architecture is superimposed, and then it looks more and more complex Up.
Let's talk about our vertical e-commerce system. After the system was launched, although the number of users was small, it was running smoothly. You have a sense of accomplishment. However, the CEO felt that the number of users was too small, so urgently mobilized operating students to do a whole network Traffic promotion. This promotion quickly brought a large wave of traffic, but at this time, the access speed of the system began to slow down.
After analyzing the log of the program, you find that the reason for the slow system is in the interaction with the database . Because the calling method of your database is to first obtain the database connection, then rely on this connection to query data from the database, and finally close the connection to release database resources . In this calling method, the connection needs to be re-established every time SQL is executed, so you wonder if it takes a long time to establish a database connection frequently and causes the problem of slow access .
So why does frequent connection creation cause slow response time? Let's look at an actual test.
I used the "tcpdump -i bond0 -nn -tttt port 4490" command to capture the network packets of online MySQL connection establishment for analysis. From the packet capture results, the entire MySQL connection process can be divided into two parts:
The first part is the first three packets. The first data packet is a "SYN" packet sent by the client to the server, the second packet is an "ACK" packet and a "SYN" packet returned by the server to the client, and the third packet is a return from the client The "ACK" packet of the server, students familiar with the TCP protocol can see that this is a TCP three-way handshake process.
The second part is the process of verifying the client password on the MySQL server. The first packet is a packet that the server sends to the client to request authentication, the second and third packets are packets that the client sends the encrypted password to the server, and the last two packets are the server back to the client Authentication OK message. From the figure, you can see that the entire connection process took about 4ms (969012-964904).
So what is the execution time of a single SQL? We have counted the SQL execution time for a period of time and found that the average execution time of SQL is about 1ms, which means that the process of establishing a connection in MySQL is more time-consuming compared to the execution of SQL. This has little effect when the request volume is small, because it takes milliseconds to establish a connection or execute SQL. However, after the amount of requests came up, if only one SQL was executed to establish a connection in the original way, only 200 database queries could be executed in 1s, and the time to establish a connection to the database accounted for 4/5.
What do you want to do at this time? After some Google searches, you find that the solution is also very simple, as long as the database connection is pre-established using the connection pool, so that there is no need to create a connection frequently when using it . After adjustment, you find that 1000 database queries can be executed in 1 second, and the query performance is greatly improved.
Use connection pool to establish database connection in advance
Although the problem was solved in a short time, you still want to thoroughly understand the core principle of solving the problem, so you start to make up lessons again. In fact, we will use many connection pools in the development process, such as database connection pool, HTTP connection pool, Redis connection pool and so on. And management is the core of the connection pool connection pool design , I will take the database connection pool, for example, to illustrate key points of connection pool management.
The database connection pool has two most important configurations: the minimum number of connections and the maximum number of connections, which control the process of obtaining connections from the connection pool:
- If the current number of connections is less than the minimum number of connections, then create a new connection to process the database request;
- If there are idle connections in the connection pool, reuse idle connections;
- If there is no connection in the free pool and the current number of connections is less than the maximum number of connections, a new connection is created to process the request;
- If the current number of connections is greater than or equal to the maximum number of connections, wait for the old connection to be available according to the time set in the configuration (the connection pool configuration of C3P0 is checkoutTimeout);
- If the waiting time exceeds this set time, an error will be thrown to the user.
You don't need to memorize this process, it's very simple. You can stop and think about how you would design if you were the designer of the connection pool, and what are the key points. This design idea will often be used in our future architecture design.
In order to facilitate your understanding of the process of sexual memory, let me give you an example. Suppose you run a small massage chair shop in the airport. There are a total of 10 massage chairs (analogous maximum number of connections). In order to save costs (massage chairs charge electricity), you usually keep 4 massage chairs in the shop. Massage chairs (minimum number of connections), the other 6 are closed. When a customer comes, if the 4 massage chairs that are normally activated are available, you can just ask him to go to the empty one. But if the 4 massage chairs are not available when the customer comes, you will start a new one until all of your 10 massage chairs are used up. What should I do after the 10 massage chairs are used up? You will tell the user, wait a while, I promise you will be available within 5 minutes (waiting time), and then the 11th user will start waiting. At this time, there will be two results: If there is a free massage chair within 5 minutes, then the customer can go directly to the free massage chair, but if the user waits for 5 minutes and is not free, then you have to Apologize and let users go to other stores to check again.
For database connection pools, according to my experience, generally online I recommend that the minimum number of connections be controlled at about 10, and the maximum number of connections should be controlled at about 20~30. Here, you need to pay attention to the maintenance of the connection in the pool, which is the massage chair I mentioned. Although some massage chairs are turned on, they sometimes fail. Under normal circumstances, the "massage chair failure" may have the following reasons:
- The IP corresponding to the database domain name has changed, and the pool connection still uses the old IP. When the database service under the old IP is closed, an error will occur when the connection query is used again;
- MySQL has a parameter "wait_timeout", which controls how long the database connection is idle before the database will actively close the connection. This mechanism is unaware of the database user, so errors will occur when we use this closed connection.
So, as the owner of a massage chair shop, how do you guarantee that the massage chair you have activated must be available?
- Start a thread to periodically check whether the connection in the connection pool is available. For example, use the connection to send a "select 1" command to the database to see if an exception will be thrown. If an exception is thrown, remove the connection from the connection pool and try shut down. At present, the C3P0 connection pool can use this method to detect whether the connection is available, which is also my preferred method.
- After obtaining the connection, first check whether the connection is available, and execute the SQL statement if it is available. For example, the testOnBorrow configuration item of the DBCP connection pool controls whether to enable this verification. This method will introduce extra overhead when obtaining a connection. Try not to enable it in the online system, and it can be used on test services.
- At this point, you have thoroughly understood the working principle of the connection pool. However, when you just wanted to breathe a sigh of relief, the CEO put forward a new requirement. You analyzed this requirement and found that in a very important interface, you need to access the database three times. Judging from experience, you think this place will definitely become a system bottleneck in the future.
Thinking further, you think you can create multiple threads to process the interaction with the database in parallel, so that the speed can be faster. However, because of the lessons learned from the last database, you think that in the high concurrency phase, the overhead of creating threads frequently will be very high, so continue to think along the lines of thought and guess the thread pool.
Pre-create threads with thread pool
Sure enough, the ThreadPoolExecutor introduced in JDK 1.5 is an implementation of a thread pool. It has two important parameters: coreThreadCount and maxThreadCount. These two parameters control the execution of the thread pool. Its execution principle is similar to the massage chair shop model we mentioned above. I will describe it to you to deepen your memory:
- If the number of threads in the thread pool is less than coreThreadCount, new threads will be created when processing new tasks;
- If the number of threads is greater than coreThreadCount, the task is dropped into a queue and executed by the currently idle thread;
- When the tasks in the queue are full, continue to create threads until it reaches maxThreadCount;
- When the number of threads reaches maxTheadCount, there are new tasks submitted, so we have to discard them.
This task processing process seems simple, but in fact there are many pitfalls, you must pay attention when using:
First of all, the thread pool implemented by JDK prioritizes placing tasks in queues for temporary storage, rather than creating more threads. It is more suitable for performing CPU-intensive tasks, that is, tasks that require a lot of CPU operations . Why is this? Because the CPU is busy when performing CPU-intensive tasks, you only need to create threads with the same number of CPU cores. More threads will switch context and reduce task execution efficiency. So when the current number of threads exceeds the number of core threads, the thread pool will not add threads, but will be placed in the queue to wait for the core threads to become free. However, the web systems we usually develop usually have a large number of IO operations, such as querying the database, querying the cache, and so on. The CPU is idle when the task is performing IO operations. At this time, if you increase the number of threads that execute the task instead of temporarily storing the task in the queue, you can perform more tasks per unit of time, which greatly improves the throughput of task execution the amount. So you see that the thread pool used by Tomcat is not the JDK native thread pool, but some transformations . When the number of threads exceeds coreThreadCount , threads will be created first until the number of threads reaches maxThreadCount , which is more suitable for a large number of IO operations in Web systems You can also refer to it in the actual application process.
Secondly, the accumulation of queues used in the thread pool is also an important indicator that we need to monitor. This indicator is particularly critical for tasks with high real-time requirements. I have encountered a weird problem in the actual project that the task has not been executed for a long time after being thrown to the thread pool. At first, I thought this was caused by a bug in the code. After investigation, it was found that the coreThreadCount and maxThreadCount settings of the thread pool were relatively small, which caused a large number of tasks to accumulate in the thread pool. The problem occurred after increasing these two parameters. It's solved. After the jump out of the pit, I put the task queue accumulation amount of important thread pool, as an important indicator of the monitoring system placed on a large screen monitor .
Finally, if you use thread pools, please remember not to use unbounded queues (that is, no fixed-size queues are set) . Maybe you feel that after using the unbounded queue, the task will never be discarded, as long as the task does not require high real-time performance, anyway, there will be a day of consumption. However, the accumulation of a large number of tasks will take up a lot of memory space. Once the memory space is full, Full GC will be triggered frequently, causing the service to be unavailable . The downtime caused by a GC that I have checked before is caused by a system in the system. The thread pool uses an unbounded queue.
Understand the key points of the thread pool, you add this feature to the system, so far, the system is stable, you successfully completed the company's research and development tasks for you.
At this time, if you look back at these two technologies, you will find that they have one thing in common: the objects they manage, whether they are connections or threads, are time-consuming to create and consume system resources. Therefore, we put them in a pool for unified management to achieve the purpose of improving performance and resource reuse.
This is a common software design idea, called pooling technology. Its core idea is space for time. It is expected to use pre-created objects to reduce the performance overhead of frequently creating objects. At the same time, the objects can be managed uniformly. Reduce the cost of using the object , in short, there are many benefits.
However, the pooling technology also has some shortcomings. For example , the objects in the storage pool must consume excess memory. If the objects are not used frequently, it will cause a waste of memory . For example, the objects in the pool need to be created in advance when the system starts, which increases the system startup time to a certain extent .
But these defects are relatively trivial compared to the advantages of pooling technology. As long as we confirm that the objects to be used are indeed time-consuming or consume resources when they are created, and these objects are indeed frequently created and destroyed, we will You can use pooling technology to optimize.
Course summary
In this lesson, I simulated the most primitive scenario of developing a vertical e-commerce system. When encountering the problem of database query performance degradation, we used the database connection pool to solve the performance problem caused by frequent connection creation, and then used the thread pool to improve Improve the performance of parallel query database.
In fact, connection pools and thread pools are not unfamiliar to you, but you may still have confusion or misunderstandings about their principles and usage. During the interview, I found that many students did not understand the basic usage of thread pools. . Borrowing this lesson, I want to emphasize again the key points:
The setting of the maximum and minimum values of the pool is very important. You can set it based on experience at the initial stage, but you still need to adjust it according to the actual operating conditions .
The objects in the pool need to be initialized before use. This is called the warm-up of the pool . For example, when using the thread pool, all core threads need to be initialized in advance. If the pool is not warmed up, it may cause more slow requests after the system restarts .
The core of the pooling technology is a practice of space-for-time optimization methods, so we should pay attention to the space occupation to avoid problems such as excessive use of space, memory leaks or frequent garbage collections .