1. redis persistence

All Redis data is stored in memory, so the data will be lost after redis restarts, so it needs to be saved to disk asynchronously from time to time (this is called "semi-persistent mode"); or every data change All are written into an append only file (aof) (this is called "full persistence mode").

Redis provides two ways of persistence, one is RDB persistence (the principle is to periodically dump Reids database records in memory to RDB persistence on disk), and the other is AOF (append only file) persistence ( The principle is to write the operation log of Reids to the file in an appended way).

2. redis persistence-RDB

Snapshots of your data can be stored at specified time intervals. (enabled by default)

2.1 Persistent configuration

Enter the command to view the configuration

vi /usr/local/redis/redis.conf

Persistence configuration of RDB

# 时间策略
save 900 1
save 300 10
save 60 10000

# 文件名称
dbfilename dump.rdb

# 文件保存路径  redis.conf所在目录下
dir ./

# 导入时是否检查
rdbchecksum yes

configuration explanation

save 900 1 means that if there is one write command within 900s, a snapshot will be triggered, which can be understood as a backup
save 300 10 means that there are 10 writes within 300s, and a snapshot will be generated

So why do you need to configure so many rules? Because the read and write requests of Redis in each time period are definitely not balanced, in order to balance performance and data security, we can freely customize when to trigger backup. So here is a reasonable configuration according to its own Redis writing situation.

Of course, if you want to disable RDB configuration, it is also very easy, just comment out save 900 1, etc., and then write in the last line of save: save ""

Modify time configuration

After modification, be sure to restart redis, enter shutdown to stop the redis server, then enter the /usr/local/redis/ directory and enter the following command to restart redis

redis-server ./redis.conf

2.2 Principle of RDB

There are two types of triggers for RDB persistence in Redis: manual triggers and Redis timing triggers.

For RDB persistence, manually enter the following command to complete manual persistence:

save: It will block the current Redis server until the persistence is complete, and it should not be used online.
bgsave: This trigger method will fork a sub-thread, and the sub-process is responsible for the persistence process, so blocking will only occur when the fork sub-process.

The scenarios that are automatically triggered mainly include the following points:

Automatically triggered according to our save mn configuration rules;
When the slave node is fully copied, the master node sends the rdb file to the slave node to complete the copy operation, and the master node will trigger bgsave;
When executing debug reload;
When executing shutdown, if aof is not enabled, it will also be triggered.

Since save is basically not used, let's focus on how the bgsave command completes RDB persistence.

Note here that the fork operation will block, resulting in a decline in Redis read and write performance. We can control the maximum memory of a single Redis instance to minimize the event consumption of Redis during fork.

3. redis persistence-AOF

Record every write operation to the server, and when the server restarts, these commands will be re-executed to restore the original data. (default off)

3.1 AOF configuration

AOF is a persistent method of redis, which is used to record all write operations, but as time increases, the aof file will become larger and larger, so it needs to be rewritten to rewrite the data in the memory in the form of commands aof file.

During the rewriting process, since redis will have new writes, in order to avoid data loss, a piece of memory will be opened up to store the write operations generated during the rewriting period. After the rewriting is completed, the data in this memory will be deleted. The operation is appended to the aof file.

# 是否开启aof
appendonly yes

# 文件名称
appendfilename "appendonly.aof"

# 同步方式
appendfsync everysec

# 重写触发配置
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

configuration explanation

appendfsync everysec actually has three modes:

always Synchronize every write command to aof immediately, very slow, but safe

everysec syncs every second, which is a compromise

no redis does not handle it and hand it over to the OS for processing. It is very fast, but it is also the least secure

In general, the everysec configuration is used, which can take into account both speed and security, and lose up to 1s of data.

auto-aof-rewrite-percentage is a parameter to set the trigger timing of aof rewrite. When the current aof file size exceeds the percentage of the aof file after the last rewrite, rewrite is triggered

For example, change 100 to 800, that is, the current aof file will be rewritten again when the current aof file exceeds 8 times of the aof file after the last rewrite, so as to ensure that the rewrite operation will not be performed again in a short period of time.
auto-aof-rewrite-min-size

Aof file rewrites the minimum file size, that is, the aof file must be triggered when it reaches this file at the beginning, and each subsequent rewrite will not be based on this variable (according to the size after the last rewrite). This variable It is only valid to initialize and start redis. If it is redis recovery, lastSize is equal to the initial aof file size.

After modification, redis needs to be restarted

Add some key-value to redis

Then check if there is an appendonly.aof file in the redis folder

You can delete the appendonly.aof file, and then restart redis to see what the data is

3.2 Principle of AOF

Since the AOF files are all appended, the AOF files will become larger and larger as the server runs, and the AOF files that are too large will have an impact on the redis server and even the host, and it will take too long to load the AOF files that are too large when Redis restarts. Too much time, these are unfriendly, so how does Redis solve this problem? Redis introduces a rewriting mechanism to solve the problem of too large AOF files.

The whole process of AOF can be roughly divided into two steps. The first step is to write commands in real time (if it is configured with appendfsync everysec, there will be 1s loss), and the second step is to rewrite the aof file.

The main process of incrementally appending to the file is: command write → append to aof_buf → synchronize to aof disk. So why write to buf first and then synchronize to disk? If real-time writing to disk will bring very high disk IO, affecting the overall performance.

Redis AOF file rewriting is the process of converting the data in the Redis process into write commands and synchronizing to the new AOF file. The rewritten AOF file will occupy a smaller volume than the old AOF file. This is caused by the following reasons of:

Data that has timed out in the process is no longer written to the file
Old AOF files contain invalid commands such as del key1, hdel key2, srem keys, set a111, set a222, etc. Rewrite uses in-process data to generate directly, so that the new AOF file only retains the write command of the final data
Multiple write commands can be combined into one, such as: lpush list a, lpush list b, lpush list c can be converted into: lpush list abc. In order to prevent client buffer overflow caused by a single command being too large, for list, set, hash, zset and other types of operations, 64 elements are used as the boundary to split into multiple commands.

The rewritten AOF file is smaller, which not only saves disk space, but more importantly, the loading time of the smaller AOF file is shorter when Redis data is restored. AOF file rewriting is divided into manual triggering and automatic triggering like RDB persistence. Manual triggering directly calls the bgrewriteaof command. We will talk about this command in detail later. Automatic triggering requires us to modify the following in redis.conf configuration

auto-aof-rewrite-percentage 800

auto-aof-rewrite-min-size 64mb

auto-aof-rewrite-percentage: represents the ratio of the current AOF file space (aof_current_size) to the AOF file space (aof_base_size) after the last rewrite, the default is 100%, which is the same size
auto-aof-rewrite-min-size: Indicates the minimum size of the AOF file when running AOF rewrite, the default is 64MB, which means that the minimum size of the AOF file is 64MB to trigger rewriting

When these two conditions are met, Redis will automatically trigger AOF file rewriting. The details of AOF file rewriting are somewhat similar to RDB persistent snapshot generation. The following is the flow chart of AOF file rewriting:

During rewriting, since the main process is still responding to commands, in order to ensure the integrity of the final backup; therefore, it will still be written into the old AOF file, and if the rewriting fails, data will not be lost.
In order to write the response write information during rewriting to the new file, a buf is also reserved for the child process to prevent the newly written file from losing data.
Rewriting is to directly generate corresponding commands from the data in the current memory, and does not need to read the old AOF files for analysis and command merging.
The text protocol directly adopted by the AOF file is mainly good in compatibility, easy to add, and high in readability, which can be regarded as modification and repair.

Whether it is RDB or AOF, a temporary file is first written, and then the file replacement is completed through rename.

4. The difference between RDB and AOF

4.1 Advantages of RDB

Once used this way, your entire Redis database will only contain one file, which is perfect for file backups. For example, you might plan to archive the last 24 hours of data every hour, and also archive the last 30 days of data every day. Through such a backup strategy, once a catastrophic failure occurs in the system, we can restore it very easily.
For disaster recovery, RDB is a very good choice. Because we can easily compress a single file and then transfer it to other storage media.
Maximize performance. For the Redis service process, when it starts persistence, the only thing it needs to do is to fork out the child process, and then the child process completes the persistence work, which can greatly prevent the service process from performing IO operations.
Compared with the AOF mechanism, if the data set is large, the startup efficiency of RDB will be higher.

4.2 Disadvantages of RDB

If you want to ensure high availability of data, that is, avoid data loss to the greatest extent, then RDB will not be a good choice. Because once the system goes down before the scheduled persistence, the data that has not been written to the disk before will be lost.
Since RDB assists in the completion of data persistence through fork child processes, if the data set is large, it may cause the entire server to stop serving for hundreds of milliseconds, or even 1 second.

4.3 Advantages of AOF

This mechanism can bring higher data security, that is, data persistence. Three synchronization strategies are provided in Redis, namely synchronization every second, synchronization every modification and no synchronization. In fact, the second synchronization is also completed asynchronously, and its efficiency is also very high. The difference is that once the system goes down, the data modified within this second will be lost. And every time the synchronization is modified, we can regard it as synchronous persistence, that is, every data change that occurs will be immediately recorded to the disk. Predictably, this approach is the least efficient. As for no synchronization, there is no need to say much, I think everyone can understand it correctly.
Since this mechanism adopts the append mode for the writing operation of the log file, even if there is a downtime during the writing process, the existing content in the log file will not be destroyed. However, if we only write half of the data in this operation and the system crashes, don’t worry, before Redis starts next time, we can pass redis-check-aof (./redis-check-aof --fix appendonly. aof) tool to help us solve the problem of data consistency.
If the log is too large, Redis can automatically enable the rewrite mechanism. That is, Redis continuously writes the modified data to the old disk file in append mode, and at the same time, Redis will also create a new file to record which modification commands are executed during this period. Therefore, data security can be better guaranteed during rewrite switching.
AOF contains a clearly formatted, easy-to-understand log file for recording all modification operations. In fact, we can also complete data reconstruction through this file.

4.4 Disadvantages of AOF

For the same amount of data sets, AOF files are usually larger than RDB files. RDB restores large datasets faster than AOF.
According to different synchronization strategies, AOF is often slower than RDB in terms of operating efficiency. In short, the efficiency of the synchronization strategy per second is relatively high, and the efficiency of the synchronization disable strategy is as efficient as RDB.

The standard for choosing between the two is to see whether the system is willing to sacrifice some performance in exchange for higher cache consistency (aof), or is willing to disable backup in exchange for higher performance when writing operations are frequent, and wait until the save is run manually , and then make a backup (rdb). Rdb is more of an eventually consistent meaning.

11. redis persistence