Detailed explanation of 6K words! Redis persistence mechanism RDB, AOF, hybrid persistence

The Redis persistence mechanism belongs to the super-high-frequency interview knowledge points of back-end interviews. It is a cliché, and it takes time to master it. Even if you are not preparing for an interview, daily development needs to be used frequently.

Recently, I took the time to greatly improve the Redis persistence mechanism I wrote before, with both pictures and texts, clear and easy to understand. Share it, I hope it will help you!

Content overview:

When using the cache, we often need to persist the data in the memory, that is, write the data in the memory to the hard disk. Most of the reasons are to reuse data later (such as restarting the machine, recovering data after a machine failure), or for data synchronization (such as the master and slave nodes of the Redis cluster synchronize data through RDB files).

An important point that Redis is different from Memcached is that Redis supports persistence and supports three persistence methods:

• Snapshot (snapshotting, RDB)
• Append only file (append-only file, AOF)
• Hybrid persistence of RDB and AOF (new in Redis 4.0)

Official document address: redis.io/topics/pers… .

RDB persistence

What is RDB persistence?

Redis can obtain a copy of the data stored in memory at a certain point in time by creating a snapshot . After Redis creates a snapshot, the snapshot can be backed up, the snapshot can be copied to other servers to create a server copy with the same data (Redis master-slave structure, mainly used to improve Redis performance), and the snapshot can be left in place to restart the server when used.

Snapshot persistence is the default persistence method adopted by Redis, which is redis.confconfigured by default in the configuration file:

clojure

copy code

save 900 1 #在900秒(15分钟)之后，如果至少有1个key发生变化，Redis就会自动触发bgsave命令创建快照。 save 300 10 #在300秒(5分钟)之后，如果至少有10个key发生变化，Redis就会自动触发bgsave命令创建快照。 save 60 10000 #在60秒(1分钟)之后，如果至少有10000个key发生变化，Redis就会自动触发bgsave命令创建快照。

Will RDB block the main thread when creating a snapshot?

Redis provides two commands to generate RDB snapshot files:

• save: Synchronous save operation will block the Redis main thread;
• bgsave: Fork a child process, the child process will execute without blocking the Redis main thread, the default option.

The main reason for saying Redis main thread instead of main process here is that after Redis starts, it mainly completes the main work in a single-threaded manner. If you want to describe it as the Redis master process, that's fine.

AOF persistence

What is AOF persistence?

Compared with snapshot persistence, AOF persistence has better real-time performance. By default, Redis does not enable AOF (append only file) persistence (it has been enabled by default after Redis 6.0), which can be appendonlyenabled by parameters:

bash

copy code

appendonly yes

After AOF persistence is enabled, every time a command that will change the data in Redis is executed, Redis will write the command into the AOF buffer server.aof_buf, and then write it into the AOF file (at this time, the system kernel buffer is not synchronized to the disk), and finally fsyncdecide when to synchronize the data in the system kernel cache to the hard disk according to the configuration of the persistence method ( strategy).

Only when it is synchronized to the disk can it be regarded as persistent storage, otherwise there is still a risk of data loss. For example, if the data in the system kernel buffer area has not been synchronized and the disk machine goes down, then this part of the data will be lost.

The save location of the AOF file is the same as that of the RDB file, both of which are dirset by parameters, and the default file name is appendonly.aof.

What is the basic process of AOF work?

The implementation of AOF persistence function can be simply divided into 5 steps:

1. Command append (append) : All write commands will be appended to the AOF buffer.
2. File write (write) : write the data in the AOF buffer to the AOF file. This step needs to call writea function (system call), and writereturn directly after writing the data to the system kernel buffer (delayed writing). Notice! ! ! There is no synchronization to disk at this time.
3. File synchronization (fsync)fsync : The AOF buffer is synchronized to the hard disk according to the corresponding persistence method ( strategy). This step needs to call fsyncthe function (system call) fsyncto perform forced hard disk synchronization for a single file operation, and fsyncit will block until the write to disk is completed and return, ensuring data persistence.
4. File rewrite (rewrite) : As the AOF file becomes larger and larger, it is necessary to rewrite the AOF file periodically to achieve the purpose of compression.
5. Restart loading (load) : When Redis restarts, the AOF file can be loaded for data recovery.

The Linux system directly provides some functions for accessing and controlling files and devices, and these functions are called system calls (syscall) .

Here is another explanation of some of the Linux system calls mentioned above:

• write: Return directly after writing to the system kernel buffer (just write to the buffer), and will not be synchronized to the hard disk immediately. While improving efficiency, it also brings the risk of data loss. Synchronous hard disk operation usually depends on the system scheduling mechanism, the Linux kernel usually synchronizes every 30s, and the specific value depends on the amount of written data and the state of the I/O buffer.
• fsync: fsyncIt is used to forcibly refresh the system kernel buffer (synchronized to the disk), to ensure that the disk write operation is completed before returning.

The AOF workflow flow chart is as follows:

What are the AOF persistence methods?

There are three different AOF persistence methods ( fsyncstrategies) in the Redis configuration file, which are:

1. appendfsync always: After the main thread calls writethe write operation, the background thread ( aof_fsyncthread) will immediately call fsyncthe function to synchronize the AOF file (swipe the disk), and fsyncthe thread will return after completion, which will seriously reduce the performance of Redis ( write+ fsync).
2. appendfsync everysec: The main thread calls and writereturns immediately after performing the write operation, and the background thread ( aof_fsyncthread) calls fsyncthe function (system call) every second to synchronize the AOF file ( write+ fsync, fsyncthe interval is 1 second)
3. appendfsync no: Called by the main thread writeReturn immediately after performing the write operation, let the operating system decide when to synchronize, generally every 30 seconds under Linux ( writebut no fsync, fsyncthe timing is determined by the operating system).

It can be seen that the main difference between these three persistence methods lies in fsyncthe timing of synchronizing AOF files (disk brushing) .

In order to balance data and write performance, you can consider appendfsync everysecthe option to let Redis synchronize AOF files once per second, and Redis performance will be less affected. And in this way, even if the system crashes, the user will only lose the data generated within one second at most. When the hard disk is busy performing write operations, Redis will also gracefully slow down its speed to adapt to the maximum writing speed of the hard disk.

Starting from Redis 7.0.0, Redis uses the Multi Part AOF mechanism. As the name implies, Multi Part AOF is to split the original single AOF file into multiple AOF files. In Multi Part AOF, AOF files are divided into three types, namely:

• BASE: Indicates the basic AOF file, which is generally generated by subprocesses through rewriting, and there is only one file at most.
• INCR: Indicates the incremental AOF file, which is generally created when AOFRW starts to execute, and there may be multiple files.
• HISTORY: Indicates the historical AOF file, which is changed from BASE and INCR AOF. Every time AOFRW is successfully completed, the corresponding BASE and INCR AOF before this AOFRW will become HISTORY, and the AOF of HISTORY type will be automatically deleted by Redis.

Multi Part AOF is not the key point, just understand it. For a detailed introduction, you can read the article on the design and implementation of Redis 7.0 Multi Part AOF by Alibaba developers .

Related issue : Redis AOF method #783 .

Why does AOF record the log after executing the command?

Relational databases (such as MySQL) usually record logs before executing commands (to facilitate failure recovery), while the Redis AOF persistence mechanism records logs after executing commands.

Why is the log recorded after the command is executed?

• To avoid additional checking overhead, AOF logging will not perform syntax checking on commands;
• Recording after the command is executed will not block the current command execution.

This also brings risks (I also mentioned it when I introduced AOF persistence):

• If Redis crashes just after executing the command, the corresponding modification will be lost;
• It may block the execution of other subsequent commands (AOF logging is performed in the Redis main thread).

Do you understand AOF rewriting?

When AOF becomes too large, Redis can automatically rewrite AOF in the background to generate a new AOF file. This new AOF file is the same as the database state saved by the original AOF file, but smaller in size.

AOF rewrite (rewrite) is an ambiguous name. This function is realized by reading the key-value pairs in the database. The program does not need to perform any reading, analysis or writing operations on the existing AOF files.

Since AOF rewriting will perform a large number of write operations, in order to avoid affecting Redis' normal processing of command requests, Redis puts the AOF rewriting program in a sub-process for execution.

During AOF file rewriting, Redis also maintains an AOF rewriting buffer , which records all write commands executed by the server during the child process creating a new AOF file. When the child process completes the work of creating a new AOF file, the server will append all the content in the rewrite buffer to the end of the new AOF file, so that the database state saved in the new AOF file is consistent with the existing database state. Finally, the server replaces the old AOF file with the new AOF file to complete the AOF file rewriting operation.

To enable the AOF rewrite function, you can call BGREWRITEAOFthe command to execute it manually, or you can set the following two configuration items to let the program automatically determine the trigger timing:

• auto-aof-rewrite-min-size: If the AOF file size is smaller than this value, AOF rewriting will not be triggered. The default value is 64 MB;
• auto-aof-rewrite-percentage: When performing AOF rewriting, the ratio of the current AOF size (aof_current_size) to the previous AOF size (aof_base_size) when rewriting. If the current AOF file size increases by this percentage value, AOF rewriting will be triggered. Setting this value to 0 will disable automatic AOF rewriting. The default value is 100.

Before Redis version 7.0, if there are write commands during rewrite, AOF may use a lot of memory, and all write commands arriving during rewrite will be written to disk twice.

After Redis version 7.0, the AOF rewriting mechanism has been optimized and improved. The following paragraph is excerpted from the article Seeing the past and future of Redis from the release of Redis7.0 by Alibaba developers .

How to deal with incremental data during AOF rewriting has always been a problem. In the past, incremental data during writing needs to be kept in memory. After writing, this part of incremental data should be written into a new AOF file to ensure data integrity. . It can be seen that AOF writing will consume additional memory and disk IO, which is also the pain point of Redis AOF writing. Although many improvements have been made before, the essential problem of resource consumption has not been resolved.

Alibaba Cloud's Redis Enterprise Edition also encountered this problem at the beginning. After several iterations of internal development, the Multi-part AOF mechanism was implemented to solve it. At the same time, it was also contributed to the community and released with this 7.0 release. The specific method is to use the base (full data) + inc (incremental data) independent file storage method to completely solve the waste of memory and IO resources, and also support the preservation and management of historical AOF files. Combined with the time information in AOF files PITR can be restored by point in time (already supported by Alibaba Cloud Enterprise Edition Tair), which further enhances the data reliability of Redis and meets the needs of users for data rollback.

Related issue : Redis AOF rewrite description is inaccurate #1439 .

Do you understand the AOF verification mechanism?

The AOF verification mechanism is that Redis checks the AOF file at startup to determine whether the file is complete and whether there is any damaged or lost data. The principle of this mechanism is actually very simple, it is to verify the AOF file by using a number called checksum (checksum) . This checksum is a number calculated by the CRC64 algorithm on the entire AOF file content. If the content of the file changes, the checksum will change accordingly. Therefore, when Redis starts, it will compare the calculated checksum with the checksum saved at the end of the file (the content of the last line saving the checksum will be ignored when calculating), so as to judge whether the AOF file is complete. If it finds a problem with the file, Redis will refuse to start and provide an appropriate error message. The AOF verification mechanism is very simple and effective, which can improve the reliability of Redis data.

Similarly, the RDB file also has a similar verification mechanism to ensure the correctness of the RDB file, which will not be repeated here.

What optimizations has Redis 4.0 made for the persistence mechanism?

Since RDB and AOF have their own advantages, Redis 4.0 began to support the hybrid persistence of RDB and AOF (closed by default, and can be aof-use-rdb-preambleenabled through configuration items).

If hybrid persistence is turned on, when AOF is rewritten, the content of RDB will be written directly to the beginning of the AOF file. The advantage of this is that it can combine the advantages of RDB and AOF, fast loading and avoid losing too much data. Of course, there are also disadvantages. The RDB part in AOF is in compressed format and is no longer in AOF format, so the readability is poor.

Official document address: redis.io/topics/pers…

How to choose RDB and AOF?

Regarding the advantages and disadvantages of RDB and AOF, the official website also gives a more detailed description of Redis persistence . Here is a brief summary based on my own understanding.

RDB is better than AOF :

• The content stored in the RDB file is compressed binary data, which saves a data set at a certain point in time. The file is small and suitable for data backup and disaster recovery. The AOF file stores every write command, similar to MySQL's binlog log, and is usually much larger than the RDB file. When AOF becomes too large, Redis can automatically rewrite AOF in the background. The new AOF file saves the same database state as the original AOF file, but the size is smaller. However, before Redis version 7.0, if there are write commands during rewrite, AOF may use a lot of memory, and all write commands arriving during rewrite will be written to disk twice.
• Use RDB files to restore data, just parse and restore the data directly, without executing commands one by one, the speed is very fast. However, AOF needs to execute each write command in turn, which is very slow. That is to say, compared with AOF, RDB is faster when restoring large data sets.

AOF is better than RDB :

• The data security of RDB is not as good as AOF, and there is no way to persist data in real time or at the second level. The process of generating the RDB file is relatively heavy. Although the work of the BGSAVE child process writing the RDB file will not block the main thread, it will have an impact on the CPU resources and memory resources of the machine. Downtime. AOF supports second-level data loss (depending on the fsync strategy, if it is everysec, the data will be lost for at most 1 second), it is only to append commands to the AOF file, and the operation is light.
• RDB files are saved in a specific binary format, and there are multiple versions of RDB in the evolution of Redis versions, so there is a problem that the old version of Redis service is not compatible with the new version of RDB format.
• AOF contains logs of all operations in an easy to understand and parse format. You can easily export AOF files for analysis, and you can also directly manipulate AOF files to solve some problems. For example, if the execution FLUSHALLcommand accidentally refreshes all content, as long as the AOF file has not been rewritten, delete the latest command and restart to restore the previous state.

In summary :

• If the data saved by Redis is lost and it doesn't matter, you can choose to use RDB.
• AOF alone is not recommended, as creating an RDB snapshot from time to time allows database backups, faster restarts, and resolves AOF engine errors.
• If the saved data requires high security, it is recommended to enable RDB and AOF persistence at the same time or enable RDB and AOF hybrid persistence.