Nginx related video analysis:
Memory pool and thread pool analysis of nginx source code Analysis of
classic problems in interviews with nginx source code
Where is the nginx module used? Take you to the realization of nginx module development
Nginx solves the c10k problem very well by using multiplexing IO (such as Linux's epoll, FreeBSD's kqueue, etc.), but the premise is that Nginx requests cannot have blocking operations, otherwise it will cause the entire Nginx process to stop serving.
However, blocking operations are inevitable in many cases. For example, when a client requests a static file, the disk IO may cause the process to block, which will cause the performance of Nginx to decrease. To solve this problem, Nginx implemented a thread pool mechanism in version 1.7.11.
Below we will analyze how Nginx solves the blocking operation problem through the thread pool.
Enable thread pool function
To use the thread pool function, you first need to add the following configuration items in the configuration file:
location / {
root /html;
thread_pool default threads=32 max_queue=65536;
aio threads=default;
}
A thread pool named "default" is defined above, containing 32 threads, and the task queue supports up to 65536 requests. If the task queue is overloaded, Nginx will output the following error log and reject the request:
thread pool “default” queue overflow: N tasks waiting
If the above error occurs, it means that the load of the thread pool is very high, which can be solved by adding the number of threads. After reaching the maximum processing power of the machine, increasing the number of threads does not improve this problem.
Everything starts from the "source"
The following mainly analyzes the source code of Nginx to understand the implementation principle of the thread pool mechanism. Now let's first understand the two important data structures ngx_thread_pool_t and ngx_thread_task_t of Nginx thread pool.
ngx_thread_pool_t结构体
struct ngx_thread_pool_s {
ngx_thread_mutex_t mtx;
ngx_thread_pool_queue_t queue;
ngx_int_t waiting;
ngx_thread_cond_t cond;
ngx_log_t *log;
ngx_str_t name;
ngx_uint_t threads;
ngx_int_t max_queue;
u_char *file;
ngx_uint_t line;
};
The purpose of each field is explained below:
-
mtx: Mutex lock, used to lock the task queue to avoid competition.
-
queue: task queue.
-
waiting: How many tasks are waiting to be processed.
-
cond: Used to notify the thread pool that there are tasks to be processed.
-
name: The name of the thread pool.
-
threads: How many threads the thread pool consists of (number of threads).
-
max_queue: The maximum number of tasks that the thread pool can handle.
ngx_thread_task_t structure
struct ngx_thread_task_s {
ngx_thread_task_t *next;
ngx_uint_t id;
void *ctx;
void (*handler)(void *data, ngx_log_t *log);
ngx_event_t event;
};
The purpose of each field is explained below:
-
next: point to the next task.
-
id: task ID.
-
ctx: the context of the task.
-
handler: The function handle for processing the task.
-
event: The event object associated with the task (when the thread pool is processed into a task, the handler callback function of the event object will be called by the main thread).
[Article benefits] C/C++ Linux server architect learning materials plus group 812855908 (data including C/C++, Linux, golang technology, Nginx, ZeroMQ, MySQL, Redis, fastdfs, MongoDB, ZK, streaming media, CDN, P2P, K8S, Docker, TCP/IP, coroutine, DPDK, ffmpeg, etc.)
Thread pool initialization
The following describes the initialization process of the thread pool.
When Nginx starts, it first calls the ngx_thread_pool_init_worker() function to initialize the thread pool. The ngx_thread_pool_init_worker() function will eventually call ngx_thread_pool_init(). The source code is as follows:
static ngx_int_t
ngx_thread_pool_init(ngx_thread_pool_t *tp, ngx_log_t *log, ngx_pool_t *pool)
{
...
for (n = 0; n < tp->threads; n++) {
err = pthread_create(&tid, &attr, ngx_thread_pool_cycle, tp);
if (err) {
ngx_log_error(NGX_LOG_ALERT, log, err,
"pthread_create() failed");
return NGX_ERROR;
}
}
...
return NGX_OK;
}
ngx_thread_pool_init()最终调用pthread_create()函数创建线程池中的工作线程，工作线程会从ngx_thread_pool_cycle()函数开始执行。
ngx_thread_pool_cycle()函数源码如下：
static void *
ngx_thread_pool_cycle(void *data)
{
...
for ( ;; ) {
if (ngx_thread_mutex_lock(&tp->mtx, tp->log) != NGX_OK) {
return NULL;
}
tp->waiting--;
while (tp->queue.first == NULL) {
if (ngx_thread_cond_wait(&tp->cond, &tp->mtx, tp->log)
!= NGX_OK)
{
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NULL;
}
}
// 获取一个任务对象
task = tp->queue.first;
tp->queue.first = task->next;
if (tp->queue.first == NULL) {
tp->queue.last = &tp->queue.first;
}
if (ngx_thread_mutex_unlock(&tp->mtx, tp->log) != NGX_OK) {
return NULL;
}
// 处理任务
task->handler(task->ctx, tp->log);
task->next = NULL;
ngx_spinlock(&ngx_thread_pool_done_lock, 1, 2048);
// 把处理完的任务放置到完成队列中
*ngx_thread_pool_done.last = task;
ngx_thread_pool_done.last = &task->next;
ngx_unlock(&ngx_thread_pool_done_lock);
(void) ngx_notify(ngx_thread_pool_handler); // 通知主线程
}
}
The main job of the ngx_thread_pool_cycle() function is to obtain a task from the task queue to be processed, and then call the handler() function of the task object to process the task, and place the task in the completion queue after completion, and notify the main thread through ngx_notify().
Add task to task queue
Through the above analysis, we know how the thread pool gets tasks from the task queue and processes them. But where did the tasks in the task queue come from? Because Nginx's mission is to process client requests, you can know that tasks are generated through client requests. In other words, the task is created by the main thread (the main thread is responsible for processing client requests).
The main thread adds a task to the task queue through the ngx_thread_task_post() function, the code is as follows:
ngx_int_t
ngx_thread_task_post(ngx_thread_pool_t *tp, ngx_thread_task_t *task)
{
...
if (ngx_thread_mutex_lock(&tp->mtx, tp->log) != NGX_OK) {
return NGX_ERROR;
}
// 通知线程池有任务需要处理
if (ngx_thread_cond_signal(&tp->cond, tp->log) != NGX_OK) {
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NGX_ERROR;
}
// 把任务添加到任务队列中
*tp->queue.last = task;
tp->queue.last = &task->next;
tp->waiting++;
(void) ngx_thread_mutex_unlock(&tp->mtx, tp->log);
return NGX_OK;
}
The ngx_thread_task_post() function first calls ngx_thread_cond_signal() to notify the thread of the thread pool that there is a task to be processed, and then adds the task to the task queue. Some people may ask, first notify the thread pool whether there will be order problems when adding tasks to the task queue. In fact, it is no problem to do this, because as long as the main thread does not call ngx_thread_mutex_unlock() to unlock the mutex, the worker threads in the thread pool will not return from ngx_thread_cond_wait().
Finishing work
When the thread pool has processed the task, it will be placed in the completion queue (ngx_thread_pool_done), and then call ngx_notify() to notify the main thread that the task is completed. After the main thread receives the notification, it will finish the work in the event module: call task.event.handler(). task.event.handler is set by the task creator, for example, in the ngx_http_copy_thread_handler() function of the ngx_http_copy_filter module:
static ngx_int_t
ngx_http_copy_thread_handler(ngx_thread_task_t *task, ngx_file_t *file)
{
...
if (tp == NULL) {
if (ngx_http_complex_value(r, clcf->thread_pool_value, &name)
!= NGX_OK)
{
return NGX_ERROR;
}
tp = ngx_thread_pool_get((ngx_cycle_t *) ngx_cycle, &name);
}
task->event.data = r;
// 设置event的回调函数
task->event.handler = ngx_http_copy_thread_event_handler;
if (ngx_thread_task_post(tp, task) != NGX_OK) {
return NGX_ERROR;
}
r->main->blocked++;
r->aio = 1;
return NGX_OK;
}
task.event.handler is set to ngx_http_copy_thread_event_handler, which means that when the task processing is completed, the main thread will call ngx_http_copy_thread_event_handler for finishing work.
Which operations will use the thread pool
Then which operations will use the thread pool to process. Generally speaking, disk IO will be processed by thread pool. In the ngx_http_copy_filter module, ngx_thread_read() will be called to read the content of the file (when the thread pool is enabled), and ngx_thread_read() will let the thread pool handle the operation of reading the file content. The code of ngx_thread_read() is as follows:
ssize_t
ngx_thread_read(ngx_thread_task_t **taskp, ngx_file_t *file, u_char *buf,
size_t size, off_t offset, ngx_pool_t *pool)
{
...
task = *taskp;
if (task == NULL) {
task = ngx_thread_task_alloc(pool, sizeof(ngx_thread_read_ctx_t));
if (task == NULL) {
return NGX_ERROR;
}
task->handler = ngx_thread_read_handler;
*taskp = task;
}
ctx = task->ctx;
...
ctx->fd = file->fd;
ctx->buf = buf;
ctx->size = size;
ctx->offset = offset;
if (file->thread_handler(task, file) != NGX_OK) {
return NGX_ERROR;
}
return NGX_AGAIN;
}
From the above code, we can see that the task handler is set to ngx_thread_read_handler, which means that ngx_thread_read_handler() will be called in the thread pool to read the contents of the file. And file->thread_handler() will call ngx_thread_task_post(). As analyzed earlier, ngx_thread_task_post() will add the task to the task queue.
Diagram
Finally, use a picture to explain the principle of Nginx thread pool mechanism.
