Storage mainly includes 6 types of threads
storage thread stack
gdb attach storage process ID can see the stack of each thread:
accept thread
The main thread is the accept thread
nio thread
nio is network IO, mainly responsible for network IO processing
The nio thread is created in storage_service_init, work_thread_entrance
The default number of nio threads is 4, you can see To thread 2 to 5 the stack stops at epoll_wait
Reporting thread
Thread 6 is tracker_report_thread_entrance, which is used as storage to report information to the tracker and obtain other storage information.
Scheduling thread
Thread 7 is the scheduling thread, sched_thread_entrance, which is mainly used to schedule some tasks regularly, such as writing the binlog buffer to the binlog file regularly.
This thread also includes log synchronization and time wheel timer.
dio thread
dio is data IO, mainly used to read and write files.
The stacks of thread 8 and thread 9 are the same, both are dio threads, dio_thread_entrance.
According to the default configuration file of storage, one thread is for reading the disk, and the other is the thread for writing the disk. By default, reading and writing are separated.
Main processes and communication of storage 5 threads
This article will not introduce the reporting thread, but mainly introduces the other five types of threads. Taking uploading files as an example, the flow chart of each thread is as follows:< /span>
accept thread
The number of accept threads can be configured. There is only one by default. The main thread is the accept thread. It is mainly used to accept connections and then take a fast_task_info object from the idle task queue (object pool).
The object pool is pre-allocated in storage_service_init according to the number of threads:
bytes = sizeof(struct storage_nio_thread_data) * g_work_threads;
g_nio_thread_data = (struct storage_nio_thread_data *)malloc(bytes);
fast_task_info is the data corresponding to each client (fd), including read and write buffers, and a pair of pipes for communication between other threads and nio threads. The nio thread is bound to the read end of the pipe.
When the client connects to storage, the accept thread writes the address of the fast_task_info object into the pipe, and then the nio thread calls back storage_recv_notify_read. When the client is connected, the initialization function storage_nio_init is called, which is to register the epoll read event.
nio thread
The nio thread, commonly known as the worker thread, is mainly responsible for network IO. The core function is epoll_wait. Then call different callback functions based on the event type:
- client_sock_read
- client_sock_write
- storage_recv_notify_read
It can be seen that the nio thread is mainly responsible for three functions:
- Read data from client and process
- Send data to client
- Receive notifications from other threads through pipes
When the client uploads the file, the callback client_sock_read is triggered n times.
After receiving the complete package for the first time, process the task (storage_deal_task) according to the command. The command to upload the file is STORAGE_PROTO_CMD_UPLOAD_FILE. At the end of storage_write_to_file, storage_dio_queue_push will be called. Blocked_queue_push is called in storage_dio_queue_push to insert fast_task_info into the blocking queue. , later the dio thread will take the task from the blocking queue for processing.
If it is not the first time to receive a complete packet, storage_dio_queue_push is called directly.
When the file is uploaded, call client_sock_write to notify the client.
When other threads want to notify the nio thread, they will write to the pipe and trigger the callback storage_recv_notify_read.
For example, the accept thread notifies the nio thread to register a read event for fd.
The dio thread will notify the nio thread to continue reading data after writing part of the file multiple times. Why is the file written multiple times here? Because a large file cannot be processed in one IO. By default, FastDFS sets a maximum recv of 256K data at a time. Assuming that the client uploads a 1M file, >1M data (including protocol headers) will actually be sent. ), so the nio thread needs to recv 5 times and call blocked_queue_push 5 times, and the dio thread needs to fetch tasks 5 times and write data 5 times.
dio thread
The dio thread is mainly responsible for file reading and writing. By default, there is one reading thread and one writing thread.
The dio_thread_entrance function is very simple, it just takes the task from the blocking queue and executes it.
After the client uploads the file, the task function taken out is dio_write_file. First call dio_open_file to open the file, and then call fc_safe_write to write the file.
When pFileContext->offset < pFileContext->end, that is, the file is not finished writing, the nio thread is notified to continue recv data.
When the nio thread receives the data again, it will write the task to the blocking queue again, and the dio thread will fetch the task again and write the file.
When pFileContext->offset >= pFileContext->end indicates that the file has been written, the dio thread needs to update the binlog.
Every time a file is added/deleted/updated, a record will be added to the binlog, which will be used to save and synchronize files in all storages in the same group.
Storage synchronization will be introduced later. This article focuses on threads.
binlog_write_cache_buff and binlog_write_cache_len will be updated in storage_binlog_write_ex
int storage_binlog_write_ex(const int timestamp, const char op_type, \
const char *filename, const char *extra)
{
int result;
int write_ret;
if ((result=pthread_mutex_lock(&sync_thread_lock)) != 0)
{
logError("file: "__FILE__", line: %d, " \
"call pthread_mutex_lock fail, " \
"errno: %d, error info: %s", \
__LINE__, result, STRERROR(result));
}
// 更新binlog_write_cache_buff和binlog_write_cache_len
if (extra != NULL)
{
binlog_write_cache_len += sprintf(binlog_write_cache_buff + \
binlog_write_cache_len, "%d %c %s %s\n",\
timestamp, op_type, filename, extra);
}
else
{
binlog_write_cache_len += sprintf(binlog_write_cache_buff + \
binlog_write_cache_len, "%d %c %s\n", \
timestamp, op_type, filename);
}
//check if buff full
if (SYNC_BINLOG_WRITE_BUFF_SIZE - binlog_write_cache_len < 256)
{
write_ret = storage_binlog_fsync(false); //sync to disk
}
else
{
write_ret = 0;
}
if ((result=pthread_mutex_unlock(&sync_thread_lock)) != 0)
{
logError("file: "__FILE__", line: %d, " \
"call pthread_mutex_unlock fail, " \
"errno: %d, error info: %s", \
__LINE__, result, STRERROR(result));
}
return write_ret;
}
Note that a synchronization thread lock is added here, because both the dio thread and the scheduling thread will access the binlog buffer.
binlog.000 file format:
timestamp + operation type + file ID
After the binlog buffer is updated and the file upload is completed, storage_nio_notify is called to write the pipeline to notify the client.
Scheduling thread
The main functions of scheduling threads are:
- Use time wheel timer to handle timeout tasks
- Loop through registered scheduled tasks
Log synchronization, binglog synchronization (fdfs_binlog_sync_func), status file synchronization, trunk binlog synchronization and other scheduling tasks are registered in setupSchedules.
From the above, we know that the dio thread will write to the binlog buffer after processing the uploaded file. At this time, binlog_write_cache_len increases. When the scheduling thread calls fdfs_binlog_sync_func again, binlog_write_cache_len > 0 will be satisfied, and then storage_binlog_fsync will be called.
int fdfs_binlog_sync_func(void *args)
{
if (binlog_write_cache_len > 0)
{
return storage_binlog_fsync(true);
}
else
{
return 0;
}
}
In fdfs_binlog_sync_func, the binlog buffer is written into the binlog file to achieve persistence. After writing ++binlog_write_version, the purpose is to let the synchronization thread start synchronization.
synchronization thread
The synchronization thread is created in the reporting thread tracker_report_thread_entrance. Assuming there are n storages, each storage will create n-1 synchronization threads to synchronize files to other storages respectively.
The synchronization thread loops to detect the binlog status. When it is detected in the storage_binlog_preread of the storage_binlog_read function that pReader->binlog_buff.version is different from binlog_write_version, the binlog file will be read.
if (pReader->binlog_buff.version == binlog_write_version &&
pReader->binlog_buff.length == 0)
{
// binlog文件没有更新则直接返回
return ENOENT;
}
When a row of binlog records is successfully extracted to StorageBinLogRecord, the return value read_result of storage_binlog_read is 0, and then storage_sync_data can be called to synchronize the file.
When synchronizing files, first call tcpsenddata_nb to send the first half of the protocol (ie, protocol header + file name, file size and other fields), and then call tcpsendfile_ex to send the file content. The core of tcpsendfile_ex is sendfile.
As for how the synchronized storage (copy) updates the files and binlog, we will talk about synchronization later.