Redis is an in-memory database, and data will disappear after a downtime. Access will hit the DB, causing a cache avalanche (a large number of keys)
to quickly restore data after Redis restarts, and provide a persistence mechanism.
Redis persistence is to quickly restore data rather than store data

Redis has two persistence methods: RDB and AOF.
Note: Redis persistence does not guarantee data integrity.

When Redis is used as a DB, the DB data must be complete, so there must be a complete data source (file, mysql)
when the system starts, and the data from this complete data source is loaded into Redis. The
amount of data is small and not easy Change, such as: dictionary library (xml, Table)

You can view information about persistence through the info command

# Persistence
loading:0
rdb_changes_since_last_save:1
rdb_bgsave_in_progress:0
rdb_last_save_time:1589363051
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:58
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

2 RDB

RDB (Redis DataBase) is the default storage method of redis, and the RDB method is completed through snapshotting .

The data at this moment
does not pay attention to the process

2.1 How to trigger a snapshot

1. Comply with the snapshot rules of the custom configuration
2. Execute the save or bgsave command
3. Execute the fluxhall command
4. Execute the master-slave copy operation (first time)

Configuration parameters are executed regularly

Configure in redis.conf: save<seconds><changes> How many seconds have the data changed

save ""  # 不使用RDB存储  不能主从
save 900 1  # 表示15分钟（900秒钟）内至少1个键被更改则进行快照。
save 300 10 # 表示5分钟（300秒）内至少10个键被更改则进行快照。
save 60 10000 # 表示1分钟内至少10000个键被更改则进行快照。

Funnel design provides performance

Command explicit trigger

Enter the bgsave command on the client.

127.0.0.1:6379> bgsave
Background saving started

2.2 RDB execution process (principle)

1. The Redis parent process first judges whether it is currently executing save, or the child process of bgsave/bgrewriteaof (aof file rewriting command), if it is executing, the bgsave command returns directly.

2. The parent process executes fork (calling the OS function to copy the main process) operation to create a child process. During this copying process, the parent process is blocked, and Redis cannot execute any commands from the client.

3. After the parent process forks, the bgsave command returns the "Background saving started" message and no longer blocks the parent process, and can respond to other commands.

4. The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion. (RDB is always complete)

5. The child process sends a signal to the parent process to indicate completion, and the parent process updates statistics.

6. After the parent process forks the child process, continue to work.

2.3 RDB file structure

1. The 5 bytes of the header are fixed as the "REDIS" string
2, and the 4 bytes "RDB" version number (not the Redis version number), currently 9, and 0009 after filling.
3. Auxiliary field, in the form of key-value

4. Store the database number
5. The dictionary size
6. The expired key
7. The main data is stored in the form of key-value
8. The end mark
9. The checksum is to see if the file is damaged or modified.
You can open the dump.rdb file with winhex to view it.

2.4 Advantages and disadvantages of RDB

Advantages
RDB is a binary compressed file, which occupies a small space and is convenient for transmission (to slaver). The
main process forks the child process. It can maximize the Redis performance. The main process cannot be too large, the amount of data in Redis cannot be too large, and the main process is blocked during the copy process.

Disadvantages
Data integrity is not guaranteed, and all data changed since the last snapshot will be lost

3 AOF

AOF (append only file) is another persistence method of Redis. Redis is not turned on by default. After enabling AOF persistence

Redis records all the commands (and their parameters ) (RESP) that have been written to the database to the AOF file to achieve the purpose of recording the database status.

In this way, when Redis is restarted, as long as these commands are played back in order, it will be restored to the original state.

AOF will record the process, RDB only cares about the result

3.1 AOF persistence implementation

Configure redis.conf

# 可以通过修改redis.conf配置文件中的appendonly参数开启
appendonly yes 

# AOF文件的保存位置和RDB文件的位置相同，都是通过dir参数设置的。
dir  ./ 

# 默认的文件名是appendonly.aof，可以通过appendfilename参数修改
appendfilename  appendonly.aof

3.2 AOF principle

The AOF file stores redis commands. The entire process of synchronizing commands to AOF files can be divided into three stages:

Command propagation: Redis sends the executed commands, command parameters, and the number of command parameters to the AOF program.

Cache addition: The AOF program converts the command to the format of the network communication protocol according to the received command data, and then appends the protocol content to the server's AOF cache.

File writing and saving: The content in the AOF cache is written to the end of the AOF file. If the set AOF saving condition is met, the fsync function or fdatasync function will be called to actually save the written content to the disk .

3.2.1 Command propagation

When a Redis client needs to execute a command, it sends the protocol text to the Redis server through a network connection. After the server receives the client's request, it will select the appropriate command function according to the content of the protocol text, and convert each parameter from the string text to the Redis string object (StringObject). Whenever the command function is successfully executed, the command parameters will be propagated to the AOF program.

3.2.2 Cache Append

After the command is propagated to the AOF program, the program will convert the command from the string object back to the original protocol text according to the command and its parameters. After the protocol text is generated, it will be appended to the end of aof_buf in the redis.h/redisServer structure.

The redisServer structure maintains the state of the Redis server, and the aof_buf field holds all the protocol text (RESP) waiting to be written to the AOF file.

3.2.3 File writing and saving

Whenever the server routine task function is executed, or the event handler is executed, the aof.c/flushAppendOnlyFile function will be called, this function performs the following two tasks:
WRITE: According to the conditions, write the buffer in aof_buf to the AOF file .
SAVE: According to conditions, call the OS function of fsync or fdatasync to save the AOF file to the disk.

3.3 AOF save mode

Redis currently supports three AOF saving modes, they are:

AOF_FSYNC_NO: Do not save.

AOF_FSYNC_EVERYSEC: Save once every second. (default)

AOF_FSYNC_ALWAYS: Save every time a command is executed. (Not recommended)

The following three subsections will discuss these three saving modes respectively.

Do not save
In this mode, WRITE will be executed every time the flushAppendOnlyFile function is called, but SAVE will be skipped.
SAVE will only be executed in any of the following situations:

Redis is shut down

AOF function is turned off

The write cache of the system is refreshed (maybe the cache is full, or the regular save operation is executed)

The SAVE operation in these three cases will cause the main Redis process to block.

Save every second (recommended)
In this mode, SAVE will be executed every second in principle, because the SAVE operation is called by a background child thread (fork), so it will not cause the main server process block.

Save every time a command is executed
In this mode, after every command is executed, WRITE and SAVE will be executed.
In addition, because SAVE is executed by the Redis main process, the main process will be blocked during the execution of SAVE and cannot accept command requests.

The impact of AOF saving mode on performance and security
For the three AOF saving modes, their blocking of the main server process is as follows:

3.4 AOF rewrite, trigger mode, hybrid persistence

The changing process of AOF recorded data is getting bigger and bigger, and it needs to be rewritten to "slim"

Redis can automatically rewrite AOF in the background (Fork subprocess) when the AOF volume becomes too large. The rewritten new AOF file contains the minimum set of commands needed to restore the current data set. The so-called "rewriting" is actually an ambiguous term. In fact, AOF rewriting does not require any writing or reading of the original AOF file. It is aimed at the current value of the key in the database.

An example is as follows:
set s1 11
set s1 22 -------> set s1 33
set s1 33

Without optimization:
set s1 11
set s1 22
set s1 33

After optimization:
set s1 33

lpush list1 1 2 3

lpush list1 4 5 6 -------- > list1 1 2 3 4 5 6

After optimization,
lpush list1 1 2 3 4 5 6

Redis does not want AOF rewriting to cause the server to be unable to process the request, so Redis decided to put the AOF rewriting program in the (background) subprocess for execution. The biggest advantage of this processing is:

1. During the AOF rewriting of the child process, the main process can continue to process command requests.

2. The child process has a copy of the data of the main process, and the child process is used instead of the thread to ensure the security of the data while avoiding locks.

However, there is a problem that needs to be solved when using a subprocess: because the subprocess is performing AOF rewriting, the main process still needs to continue processing commands, and new commands may modify the existing data, which will make the current database data and The data in the rewritten AOF file is inconsistent.

In order to solve this problem, Redis has added an AOF rewrite cache. This cache is enabled after the child process is forked. After receiving a new write command, the main Redis process will append the protocol content of the write command to the existing In addition to the AOF file, it will also be appended to this cache.

3.4.1 Analysis of the rewriting process (the entire rewriting operation is absolutely safe):

In the process of creating a new AOF file, Redis will continue to append commands to the existing AOF file. Even if a shutdown occurs during the rewriting process, the existing AOF file will not be lost. Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending the new AOF file.

When the child process is performing AOF rewriting, the main process needs to perform the following three tasks:

Process the command request.

Append the write command to the existing AOF file.

Append the write command to the AOF rewrite cache.

This way you can guarantee:

The existing AOF function will continue to execute, even if there is a downtime during AOF rewriting, there will be no data loss. All commands to modify the database will be recorded in the AOF rewrite cache. When the child process completes the AOF rewriting, it will send a completion signal to the parent process. After receiving the completion signal, the parent process will call a signal processing function and complete the following tasks:

1. Write all the contents in the AOF rewrite cache to the new AOF file.

2. Rename the new AOF file to overwrite the original AOF file.

Commands in the +AOF rewriting process in the Redis database------->new AOF file---->overwrite the old

When step 1 is completed, the status of the existing AOF file, the new AOF file and the database are completely consistent.

When step 2 is completed, the program completes the alternation of the new and old AOF files.

After this signal processing function is executed, the main process can continue to accept command requests as usual. In the entire AOF background rewriting process, only the final write cache and rename operation will cause the main process to block. At other times, AOF background rewriting will not block the main process, which will cause AOF rewriting to performance The impact is minimized.

The above is the AOF background rewrite, that is, the working principle of the BGREWRITEAOF command (AOF rewrite).

3.4.2 Trigger mode

1. Configure trigger configuration
in redis.conf

# 表示当前aof文件大小超过上一次aof文件大小的百分之多少的时候会进行重写。如果之前没有重写过，以启动时aof文件大小为准
auto-aof-rewrite-percentage 100

# 限制允许重写最小aof文件大小，也就是文件大小小于64mb的时候，不需要进行优化
auto-aof-rewrite-min-size 64mb

2. Execute the bgrewriteaof command

127.0.0.1:6379> bgrewriteaof
Background append only file rewriting started

3.4.3 Hybrid persistence

RDB and AOF have their own advantages and disadvantages. Redis 4.0 began to support the hybrid persistence of RDB and AOF. If the mixed persistence is turned on, the content of the rdb will be written directly to the beginning of the aof file during aof rewrite.

RDB head + AOF body---->appendonly.aof

Turn on hybrid persistence

aof-use-rdb-preamble yes

We can see that the AOF file is the header of the RDB file and the content of the AOF format. When loading, it will first identify whether the AOF file starts with the REDIS string. If it is, it will be loaded in the RDB format. After loading the RDB, continue to press the AOF format. Load the rest.

3.4.4 Loading and data restoration of AOF files

Because the AOF file contains all the write commands needed to rebuild the database state, the server only needs to read in and re-execute the write commands saved in the AOF file to restore the database state before the server shuts down. Redis reads the AOF file and restores the database. The detailed steps of the status are as follows:

1. Create a fake client without a network connection: Because Redis commands can only be executed in the client context, and the commands used when loading the AOF file come directly from the AOF file instead of the network connection. Therefore, the server uses a pseudo client without a network connection to execute the write command saved in the AOF file. The effect of the pseudo client executing the command is exactly the same as that of the client with a network connection.

2. Analyze and read a write command from the AOF file

3. Use the pseudo client to execute the read command

4. Continue to execute steps 2 and 3 until all write commands in the AOF file have been processed

After completing the above steps, the database state saved in the AOF file will be completely restored. The whole process is shown in the following figure:

4 Comparison of RDB and AOF

1. RDB stores a snapshot of data at a certain moment, using binary compression storage, AOF storage operation commands, using text storage (hybrid)
2. RDB performance is high, AOF performance is low
3. RDB will lose its configuration trigger state after the last snapshot All the changed data, AOF is set to save once per second, the data will be lost at most 2 seconds
4. Redis runs in master server mode, RDB does not save expired key-value pair data, Redis runs in slave server mode, RDB will save expired data Key-value pairs, when the master server synchronizes with the slave server, then clear the expired key-value pairs.

When AOF is written to a file, a del command will be added to the expired key. When AOF rewrite is performed, the expired key and del command will be ignored.

5 Application scenarios

In-memory database rdb+aof data is not easy to lose
. Original data source: It is initialized from the original data source every time it starts, so there is no need to open persistence (small amount of data). The
cache server RDB generally has high performance

When data is restored, if
there is rdb+aof, then aof will be restored, because RDB will cause the loss of files, and the relative data of AOF must be complete.
Only rdb, restore rdb

Pull hook configuration strategy

Pursue high performance: restore from data source without redis downtime.
Dictionary library: do not expel, and ensure data integrity without persistence

Used as a DB, the main
and subordinate data volume is not small, and the cache performance is higher: open the RDB
Redis data volume storage is too large, the performance drops suddenly, and the
fork time is too long to block the main process,
only open AOF

4.1.2 Redis persistence, reasons, RDB method (trigger, principle, structure, advantages and disadvantages), AOF method (principle, save mode, rewrite, trigger method, hybrid persistence), RDB/AOF comparison, application scenarios

Redis persistence

1. Why persist