Introduction
The data in Redis is stored in the memory. If it suddenly goes down, all the data in the memory will be lost. If the data can be recovered from the back-end database, if the data only exists in Redis, then all the data will be lost. And if there are a lot of requests, the pressure on the MySQL server will be great.
So the best way is to persist the data and recover quickly when it goes down
There are two persistence methods in Redis, rdb snapshot and aof log
RDB
rdb is to take a snapshot of the current database state and save the data of a certain stage through a binary file. You can analogize photography. The more data in the memory, the longer it takes to generate a snapshot, and the longer it takes to write the snapshot to disk.
At this time, we don't have to ask, will generating a snapshot block the main thread? If the main thread is blocked, it will affect the processing of normal requests
There are two commands in Redis that can be used to generate RDB files, one is save and the other is bgsave
- save: execute in the main thread, will cause blocking
- bgsave: The main thread forks a child process responsible for creating the RDB file, and does not block the main thread
We certainly did not hesitate to choose bgsave, after all, it will not block the main thread
Then when we use bgsave to generate a mirror, can the data be modified?
If the data is allowed to be modified, there will be many problems. For example, the bgsave child process has just finished persisting a key, and as a result, the main thread deletes the key, which will cause inconsistent data.
If the data is not allowed to be modified, then all write operations can only be performed until the rdb file is generated, which affects performance.
At this time we have to mention COW, redis uses a multi-process COW mechanism to achieve snapshot persistence
Copy-On-Write，COW
When Redis is persisting, it forks a child process, and the snapshot persistence is handed over to the child process to complete. When the child process just spawns, it shares the data segment and code segment inside with the parent process. So at the moment of process separation, there is no change in the chance of memory growth.
The child process does not modify the data in the memory for persistence, but the main thread is different, it will persistently receive the client's modification request, and then modify the data in the memory.
At this time, the COW mechanism of the operating system is used to separate the data segment pages. The data segment is composed of pages of many operating systems. When the parent process modifies the data of one of the pages, the shared page will be copied and separated, and then the copied page will be modified. This is whether the corresponding page of the child process has not changed, or the data when the process is generated.
As the modification operation of the parent process progresses, more and more shared pages are separated, and the pages will continue to grow, but not more than twice the original memory.
The data in the child process has not changed, and it can be persisted with peace of mind.
If a snapshot is generated every 1 minute, the operations performed after the snapshot was generated will still be lost after the downtime (up to operations within 1 minute). We shorten the time to generate snapshots, which will affect Redis performance. After all, the fork child process will block the main thread and frequently read and write the disk, which will also put a lot of pressure on the disk.
This is to have to mention another way of persistence, aof log
AOF
When we execute a command every time, the corresponding operation is recorded in the aof log. When redis is down, we can recover the data by simply replaying the log. And Redis saves aof logs in the form of text
For example, when we execute the following command
set key value
The following content will be appended to the aof file
*3
$3
set
$3
key
$5
value
*3 means that the current command has 3 parts, each part starts with "$+number", the number indicates the command, and the key or value consists of several bytes
It should be noted that what is recorded in redis is the post-write log , that is, execute the command first, and then write the log.
If the command is executed successfully, there is no time to write the log? Then this command is lost after the service is down?
It should be that the AOF log is written in the main thread. If the log is written to the disk every time, wouldn't it affect performance?
Fortunately, redis provides us with three ways to write AOF logs
always : Synchronous write back, synchronize to disk
after the write command is executed everysec : write back every second, after each write command is executed, just write the log to the memory buffer of the aof file first, and the contents of the buffer every 1 second Write to disk
no : the operating system controls the write back. After each write command is executed, it just writes the log to the memory buffer of the aof file, and the operating system decides when to write the buffer content back to the disk
We can choose according to the system's requirements for high performance and reliability.
The aof log records the state of the database by saving the executed write commands. As time goes by, the aof log will become larger and larger, and the time required to restore the data using the aof file is also getting longer. Is there any optimization plan? At this time the aof log rewrite is on the scene.
AOF log rewrite
If the client executes the following 5 commands in sequence
127.0.0.1:6379> rpush list 1
(integer) 1 // [1]
127.0.0.1:6379> rpush list 2
(integer) 2 // [1, 2]
127.0.0.1:6379> rpush list 3
(integer) 3 // [1, 2, 3]
127.0.0.1:6379> lpop list
"1" // [2, 3]
127.0.0.1:6379> rpush list 1
(integer) 3 // [2, 3, 1]
To record the status of the key of list alone requires 5 logs. If only these 5 commands can be merged into rpush list 2 3 1 for this fame and fortune for you. In fact, this is what aof log rewriting is going to do, so how to achieve it?
Although Redis names the function of generating new AOF files as "aof rewriting", AOF rewriting does not require any reading and analysis operations on existing AOF files. Instead, it directly reads the latest value in the memory, and then saves the corresponding command.
For example, in the above example, redis directly reads the value of the list, and generates a rpush list 2 3 1 command and puts it in the aof log.
You can see that AOF rewriting is a very time-consuming operation, so will it block the main thread?
No, because as an optimization method, Redis certainly doesn't want it to be blocked. Therefore, the main thread forks a bgrewriteaof child process every time it is rewritten. The bgrewriteaof child process uses Copy-On-Write technology to read the data in the memory and write a new aof log
In the process of rewriting the AOF log, how should the operations performed by the main thread be written to the new AOF log?
In fact, in the process of AOF log rewriting, the main thread will synchronize the operation to the AOF buffer and AOF rewrite buffer. When the child thread completes AOF rewriting, and writes the contents of the AOF buffer to the new AOF log, it will replace the old AOF log with the new AOF log