Persistence of redis study notes (AOF&RDB)

Overview

Redis stores all data in memory. If it accidentally goes down, the data will be lost. In order to solve this problem, redis solves this problem through persistence. There are two types of persistence in redis: RDB (redisDataBase) and AOF (append only file) . Next, let's briefly understand these two persistence mechanisms.

RDB (redisDataBase)

Realization principle

RDB persistence is achieved through COW (copy on write) mechanism. A child process is generated by fork, the data is traversed and read, and then serialized and written to disk. The child process only persists the data and does not modify the existing memory data structure.

So if the parent process has data modification, how does redis deal with it?

At this time, the cow mechanism of the operating system will be used to separate the data segment pages. The data segment is composed of many operating system pages (a page 4K). When the parent process needs to modify the data of one of the pages, it will Make a copy of this copied page and modify the copied page. The page corresponding to the child process has not changed, and it is still the data before the child process was generated. In addition, due to the relatively high proportion of cold data in the redis instance, it is generally rare that all pages are separated

Note: The default use of bgsave (background persistence) has basically no effect on the use, and the other save (call rdbSave directly, blocking the main Redis process until the save is completed) affects the use

RDB advantages

RDB will generate multiple data files, and each data file represents the data of redis at a certain moment. This multiple data file method is very suitable for cold backup.
When RDB provides read and write services to redis externally, the impact is very small, because the main redis process only needs to fork a child process to allow the child process to perform RDB persistence on the disk io
RDB is faster in restoring large data sets than AOF

RDB disadvantages

If redis wants to lose as little data as possible when it fails, RDB is not as good as AOF. For example, if the snapshot is taken at 1:00, it is down when the snapshot is taken at 1:10, and 10 minutes of data will be lost at this time.
Each time RDB forks a child process to execute an RDB snapshot to generate a file, if the file is particularly large, it may cause the client to provide services to be suspended for several milliseconds or seconds

AOF(append only file)

Realization principle

AOF writes the content to the memory cache through fsync, and then the kernel writes the data to the disk asynchronously. Redis executes fsync every one second by default. This time is configurable. There are two other strategies: never call fsync (let the operating system decide when to synchronize to the disk), call it once every time a command is executed fsunc.

note:

In AOF, the command needs to be executed first to ensure that the command is executed successfully, and then the command is immediately stored in the AOF log.
Restart is to restore data through AOF log

AOF log slimming

The AOF log is continuously incrementally backed up, so it grows slowly, and many commands are repeated (for example, the set command is executed repeatedly). In redis, a bgrewirteaof command is provided to slim down the AOF log. The principle is to open a subprocess to traverse the memory, convert it into a series of redis operation commands, and serialize it into a new AOF log file. After the serialization is complete, append the incremental AOF log that occurred during the operation to a new AOF log, and replace the original AOF log after the append is completed

AOF advantages

AOF can better protect data from loss. Generally, AOF will perform an fsync operation through a thread in the background every 1 second. If the redis process hangs, up to 1 second of data will be lost.
AOF is written in an append-only mode, so there is no disk addressing overhead, and the write performance is very high.
The commands of the AOF log file are recorded in a very readable way. This is very suitable for emergency recovery from a catastrophic accidental deletion. If someone accidentally clears all data with the fluxhall command, as long as the rewrite has not been executed at this time, then it can Delete flushall in the log file and restore

AOF disadvantages

For the same file, the AOF file is larger than the RDB data snapshot.
After AOF is turned on, the QPS supported for writing will be lower than the QPS supported by RDB, because AOF is generally configured to fsync operations per second, and the fsync operations per second are still very high

Data recovery is slow and not suitable for cold backup.

Redis hybrid persistence (4.0)

Hybrid persistence is to combine RDB persistence and AOF persistence to write AOF files together. The AOF log is no longer the full log, or the incremental AOF log from the beginning to the end of persistence (generally smaller)

When restarting, the RDB part will be loaded first, and the AOF log will be replayed