This work is licensed under the Creative Commons Attribution-Non-Commercial Use-Share 4.0 International License Agreement in the same way .
This work ( Lizhao Long Bowen by Li Zhaolong creation) by Li Zhaolong confirmation, please indicate the copyright.
Article Directory
introduction
After a lapse of ten months, I once again picked up a pen to explore Redis-related things, and I was still a little excited. Recently, I plan to use two to three articles to review Redis-related things. It is not only a supplement to the missing knowledge points, but also a full stop to the review of Redis in this period of time.
This article mainly wants to talk about a question, is Redis single-threaded or multi-threaded ? I simply searched for this issue on major platforms and found that at least 70% of the articles are basically worthless, but there are also many good articles. This article is based on the discussion of the predecessors, coupled with my own understanding, to discuss this issue. However, because the source code version is too low, some parts of the discussion cannot be attached to the code.
Single thread or multi thread
When should we use multithreading? As described in [1], it is clear that there are two reasons for using multithreading, namely, the use of multi-core efficiency and separation of concerns . On the contrary, the reason for not using multithreading is that the benefits are not as good as the rewards .
To throw out the answer first, Redis uses multithreading instead of single threading . Of course, the meaning here is a bit different from the default idea in our usual chat. When we usually talk about WebServer single-threaded or multi-threaded, what we actually discuss is whether there are more or more worker threads, which is the overall thread design of Worker Is it one or more? To give the simplest example, the one loop per threadmodel used by muduo is a very classic semi-synchronous and semi-asynchronous model, which is a multi-threaded model, in which one thread handles the connection and the remaining thread handles the request. The reason for this is because WebServer (from the perspective of RabbitServer 's performance analysis) is a computationally intensive program, and we need to better apply the increase in computing power brought by multi-core.
This is not the case with Redis. It is easy to see that the computing tasks performed by the Redis server program are actually very simple. Data is received in the epoll loop, then the command is parsed, and finally executed. Because the data structure design in Redis is very clever, basically operating these data does not take too long. You can see the following text in [3]:
- It’s not very frequent that CPU becomes your bottleneck with Redis, as usually Redis is either memory or network bound. For instance, using pipelining Redis running on an average Linux system can deliver even 1 million requests per second, so if your application mainly uses O(N) or O(log(N)) commands, it is hardly going to use too much CPU.
- However, to maximize CPU usage you can start multiple instances of Redis in the same box and treat them as different servers. At some point a single box may not be enough anyway, so if you want to use multiple CPUs you can start thinking of some way to shard earlier.
- You can find more information about using multiple Redis instances in the Partitioning page.
- However with Redis 4.0 we started to make Redis more threaded. For now this is limited to deleting objects in the background, and to blocking commands implemented via Redis modules. For future releases, the plan is to make Redis more and more threaded.
However, if the amount of data is relatively large, disk IO and network IO may become the overall performance bottleneck. Because whether it is to transfer data to the client, the transmission of RDB packets during synchronization, the INFO packet in the cluster, the PING/PONG packet, the synchronization of commands in the master-slave model, the heartbeat packet, etc. are all large network IO costs, while big data The regular flushing of the AOF cache is also a huge pressure for disk IO.
And the optimization method of network IO Huan Shen has also mentioned to us. At present, there are two ways to open source, one is 协议栈优化, which represents Sina’s fastsocket ; the other is by pass kernelthe method, from the network card driver to the user-mode protocol stack. Used, it represents Intel's DPDK . Obviously these have nothing to do with how the database is played.
But we know a problem, that is, the IO of a general Gigabit network card has an upper limit. The payload of a packet on an Ethernet is generally [84, 1538] bytes. We assume that each packet is filled to the full, that is, among them The data in 1538 bytes is 1460 bytes, and the maximum flow rate of a gigabit network card per second is about 125MB, so the maximum effective data volume that a gigabit network card can withstand per second is about 118MB. What problems will occur when a thread runs network IO? It is possible that this solitary thread has encountered the following situations:
- Operation bigkey : writing to a bigkey takes more time when allocating memory. Similarly, deleting bigkey and releasing memory will also produce time-consuming
- Use commands that are too complex : such as SORT/SUNION/ZUNIONSTORE/KEYS, or O(N) commands, and N is very large. Like compressed lists, cascading operations may also occur.
- A large number of keys expired collectively : The expiration mechanism of Redis is also executed in the main thread. When a large number of keys expire in a concentrated manner, it will take time to delete expired keys when processing a request, which will take longer.
- Elimination strategy : The walking strategy is also executed in the main thread. When the memory exceeds the Redis memory limit, some keys need to be eliminated for each write, which will also cause time-consuming and longer.
- Master-slave full synchronization generates RDB : Although the fork child process is used to generate data snapshots, the fork will block the entire thread at this moment (copy on write, the entire process data will be copied), the larger the instance, the longer the blocking time.
Then the network IO is not managed by threads during this period, and it can only be read in epoll next time after processing all the tasks. If the network IO pressure is really great, the blocking time may cause the receiving buffer to be tight. , Thereby affecting the throughput of the entire application and the response time of subsequent operations. Multithreading optimizes network IO from this perspective.
Although we have discussed this result, a mature software cannot be built in one step, and the development of a project cannot be considered comprehensively at the beginning. It is always an iterative process [5]. The logic of single-threaded processing of the entire database brings the advantages of ease of development and ease of implementation. I think this is a point that Redis is a single-threaded processing that cannot be ignored.
Asynchronous operation
The reason for interjecting here is that many people tend to associate asynchrony with multithreading. Does asynchrony have to be multithreaded? Of course not necessarily.
And such single-threaded processing (worker's single thread is also counted) must use asynchronous operations, otherwise it is not a problem of low efficiency, but a problem of availability. Imagine an send/recvoperation that blocks your solitary thread for 0.5 seconds, and that's a fart. This kind of operation can be extended to many places, such as the transmission of commands in redis, connecting with a newly discovered slave server or sentinel or master server (non-blocking may take up to several seconds), or disconnecting from a socket.
In fact, the essence is that the actual execution is not when the order is issued, but at an appropriate time.
Asynchronous operations are used in many places in Redis, but only a few use multithreading. I personally think that the reason is that these operations cannot be asynchronous, no matter what the CPU time is spent, blocking is unavoidable.
Where are multithreading used
After selling so many points, where does Redis use multi-threading (process)? The following are all the information I can find based on the source code analysis of version 3.2 and search engines.
- RDB endurance
- AOF rewrite
- AOF close operation
- AOF refresh cache
- Deleting objects asynchronously
lazyfree(4.0) - Network IO and command analysis (6.0)
Of which the first two unnecessary to say, by BGSAVEand BGREWRITEAOFcan be performed in the child RDB persistence and AOF rewrite, of course, serverCronin time to meet certain conditions will trigger, do not explain in detail here.
And the third and fourth article is the key discussion object of today's article, that is, the BIO mechanism . We put the description of this problem in the next section.
As for Article 56, it is a feature introduced in the new version. Deleting objects asynchronously is easy to understand [9][10][11]. We have already discussed network IO before.
BIO mechanism
The first time I noticed this problem was when I was looking at the implementation of the AOF part of the source code, I found backgroundRewriteDoneHandlerthat there is bioCreateBackgroundJobsuch a strange function in it, its function is to asynchronously close the old AOF file that has just been executed, and it is actually running in another Thread. Two questions popped up in my head at the time:
- Why does the close need asynchronous operation?
- What else can this thread do?
The first question can be found in the comments of the source code:
- Currently there is only a single operation, that is a background close(2) system call. This is needed as when the process is the last owner of a reference to a file closing it means unlinking it, and the deletion of the file is slow, blocking the server.
- Currently, only the close(2) operation is executed in the background: because when the server is the last owner of a file, closing a file means unlinking it, and deleting the file is very slow and will block the system, so we will close(2) Put it in the background.
As for the answer to the second question, it can also be regarded as another question. What exactly did BIO do ? In fact, the full name of BIO Background I/O, not the bio[13] that represents the disk IO request, is the background IO service of Redis, which implements the function of performing work in the background. BIO is executed in multiple threads.
In fact, the implementation of this thing is very simple, it is a producer-consumer model using locks and condition variables.
Called in the redis.c/main function initServer, which is called bioInit, this is the initialization function of BIO:
void bioInit(void) {
pthread_attr_t attr;
pthread_t thread;
size_t stacksize;
int j;
// 初始化 job 队列，以及线程状态；其实也就是调用标准C库，初始化条件变量和锁，以及初始化队列头
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
pthread_mutex_init(&bio_mutex[j],NULL);
pthread_cond_init(&bio_condvar[j],NULL);
bio_jobs[j] = listCreate();
bio_pending[j] = 0;
}
// 设置栈大小;
pthread_attr_init(&attr);
pthread_attr_getstacksize(&attr,&stacksize); // 默认大小为4294967298
if (!stacksize) stacksize = 1; /* The world is full of Solaris Fixes */
while (stacksize < REDIS_THREAD_STACK_SIZE) stacksize *= 2;
pthread_attr_setstacksize(&attr, stacksize);
// 创建线程
for (j = 0; j < REDIS_BIO_NUM_OPS; j++) {
void *arg = (void*)(unsigned long) j;
if (pthread_create(&thread,&attr,bioProcessBackgroundJobs,arg) != 0) {
redisLog(REDIS_WARNING,"Fatal: Can't initialize Background Jobs.");
exit(1);
}
bio_threads[j] = thread;
}
}
Let's take a look at how to create a task:
void bioCreateBackgroundJob(int type, void *arg1, void *arg2, void *arg3) {
struct bio_job *job = zmalloc(sizeof(*job));
job->time = time(NULL);
job->arg1 = arg1;
job->arg2 = arg2;
job->arg3 = arg3;
pthread_mutex_lock(&bio_mutex[type]);
// 将新工作推入队列
listAddNodeTail(bio_jobs[type],job);
bio_pending[type]++;
pthread_cond_signal(&bio_condvar[type]);
pthread_mutex_unlock(&bio_mutex[type]);
}
A standard producer-consumer task is inserted, lock first, then signalclick.
The specific consumer code is in
#define REDIS_BIO_CLOSE_FILE 0 /* Deferred close(2) syscall. */
#define REDIS_BIO_AOF_FSYNC 1 /* Deferred AOF fsync. */
#define REDIS_BIO_NUM_OPS 2
void aof_background_fsync(int fd) {
bioCreateBackgroundJob(REDIS_BIO_AOF_FSYNC,(void*)(long)fd,NULL,NULL);
}
if (oldfd != -1) bioCreateBackgroundJob(REDIS_BIO_CLOSE_FILE,(void*)(long)oldfd,NULL,NULL);
The above code is to create two types of tasks;
The consumer-related code is as follows:
void *bioProcessBackgroundJobs(void *arg) {
struct bio_job *job;
unsigned long type = (unsigned long) arg;
sigset_t sigset;
/* Make the thread killable at any time, so that bioKillThreads()
* can work reliably. */ // 设置线程取消相关的条件[14]
pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, NULL);
pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, NULL);
pthread_mutex_lock(&bio_mutex[type]);
/* Block SIGALRM so we are sure that only the main thread will
* receive the watchdog signal. */
sigemptyset(&sigset);
sigaddset(&sigset, SIGALRM);
if (pthread_sigmask(SIG_BLOCK, &sigset, NULL)) // 设置线程掩码
redisLog(REDIS_WARNING,
"Warning: can't mask SIGALRM in bio.c thread: %s", strerror(errno));
while(1) {
listNode *ln;
/* The loop always starts with the lock hold. */
if (listLength(bio_jobs[type]) == 0) {
// 没考虑虚假唤醒啊
pthread_cond_wait(&bio_condvar[type],&bio_mutex[type]);
continue;
}
/* Pop the job from the queue.
*
* 取出（但不删除）队列中的首个任务
*/
ln = listFirst(bio_jobs[type]);
job = ln->value;
/* It is now possible to unlock the background system as we know have
* a stand alone job structure to process.*/
pthread_mutex_unlock(&bio_mutex[type]);
/* Process the job accordingly to its type. */
// 执行任务
if (type == REDIS_BIO_CLOSE_FILE) {
// 子线程中实际执行任务的代码
close((long)job->arg1);
} else if (type == REDIS_BIO_AOF_FSYNC) {
aof_fsync((long)job->arg1);
} else {
redisPanic("Wrong job type in bioProcessBackgroundJobs().");
}
zfree(job);
/* Lock again before reiterating the loop, if there are no longer
* jobs to process we'll block again in pthread_cond_wait(). */
pthread_mutex_lock(&bio_mutex[type]);
// 将执行完成的任务从队列中删除，并减少任务计数器；需要加锁
listDelNode(bio_jobs[type],ln);
bio_pending[type]--;
}
}
The main thread bioCreateBackgroundJobinserts different types of tasks into the task linked list, so this is actually a blocking queue, in which there are only two types of tasks, namely, closing old files and refreshing the AOF page cache during AOF rewriting.
#define aof_fsync fdatasync
It is worth mentioning that in the 3.2 version, the cache refresh is used fdatasync. This method is fsyncnot safe enough compared to and sync_file_rangenot efficient (but slightly safer). I don’t know why this is used. It is used in higher versions. fsync, I don’t know if the later version will be updated.
The above is the principle and implementation of BIO in version 3.2. Apart from other things, this real-life example of using producers and consumers in real life is a good learning material.
lazyfree mechanism
The description in [11] is clear enough, I don’t need to write another article to describe this problem.
We can see in [11] that lazyfreeit is actually a newly created BIO thread, which supports deleting keys, dictionaries, and key-slotstructures in the cluster (jump table).
The operation is also very simple, that is, check the incoming parameters before deleting the key. If it is an asynchronous option, call the asynchronous delete version. What it does is to seal an object and throw it into the BIO request queue.
Of course, there are other situations where the key may be deleted, so Redis 4.0 adds several new configuration options, as follows:
slave-lazy-flush：The option to clear the data after the slave receives the RDB filelazyfree-lazy-eviction: Memory full eviction optionlazyfree-lazy-expire: Expired key delete optionlazyfree-lazy-server-del: Internal delete options, such as rename may be accompanied by an implicit delete key [15].
Respectively represent whether to activate in the four deletion situations lazy free. For specific content, please refer to [15].
Network IO
The following figure comes from [8], which basically describes the application of multithreading in version 6.0.
The code analysis here can be viewed from the description in [2]. A polling method is used to make the entire process lock-free. It is indeed very clever, but the key to the problem is that the IO sub-thread and the main thread are both non-stop rounds. Inquiry without insomnia, so will it be full of CPU during idle time? Redis's current practice is to close these IO threads when waiting to process a few connections, but it feels that it is still treating the symptoms rather than the root cause.
to sum up
It is indeed a very interesting question, involving a lot of knowledge points. Later, I have the opportunity to have a deeper look at the implementation details of the 6.0 version of multithreading. It must be a very interesting experience.
reference:
- 《C++ Concurrency In Action》
- " Officially support multi-threading! Performance Comparison and Evaluation of Redis 6.0 and the Old Version "
- 《Redis FAQ》
- " Redis Basics (2) High-Performance IO Model "
- " Software Engineering Study Notes (Full) "
- 《Why Redis is Single-Threaded》
- " A little understanding of Linux server programming "
- " Detailed Explanation of Redis Multithreading Principle "
- " [Redis study notes] new features of redis4.0-non-blocking delete "
- " Walking on Thin Ice-The Great Sacrifice of Redis Lazy Delete "
- " Redis · lazyfree · The Gospel of Big Key Deletion "
- " Redis BIO System "
- " Let's talk about Linux IO again "
- 《pthread_setcancelstate》
- " Redis4.0 New Features (3)-Lazy Free "
- " Redis essay-rename efficiency problem "