Understand Redis persistence, detailed explanation and operation of the working principle of RDB mode and AOF mode

redis persistence

Although Redis is a memory-level caching program, that is, redis uses memory for data caching, but it can save the memory data to the hard disk according to a certain strategy, so as to achieve the purpose of data persistence.
Currently redis supports two types Different types of data persistence storage mechanisms, RDB and AOF

• RDB can be regarded as a snapshot of Redis at a certain moment, which is more suitable for disaster recovery. It can be regarded as a snapshot, or it can be understood as a Mysql Dump
  • AOF is the log of write operations, which can be used as MySQL Binlog to help understand

RDB mode

How RDB Mode Works

RDB (Redis DataBase): Time-based snapshots. By default, only the latest snapshot is retained. The characteristic is that the execution speed is relatively fast. The disadvantage is that the data that has not been snapshotted between the last snapshot and the current point in time may be lost

The specific process of RDB bgsave to implement snapshots:

• Redis forks a child process from the master main process, and uses the copy-on-write mechanism. The child process saves the data in the memory as a temporary file, such as: tmp-.rdb. After the data is saved, the last saved RDB file will be saved. Replace, and then close the child process, so that you can ensure that the data saved every time an RDB snapshot is complete
• Because when directly replacing the RDB file, there may be problems such as sudden power failure, and the RDB file has not been saved completely because the sudden shutdown stops saving, resulting in data loss. In the future, you can manually perform the RDB file generated each time. Backup, so you can maximize the preservation of historical data

1. Redis executes the bgsave command, Redis judges that there are currently executing child processes, such as RDB/AOF child processes, and the bgsave command returns directly
2. Fork out the child process, the Redis parent process will block during the fork operation
3. fork完成返回　　59117:M 13 Apr 13:44:40.312 * Background saving started by pid 59180
4. The child process process generates quick find files for memory data
5. The child process tells the parent process to complete the processing
   

Realize the RDB way

• save synchronization, will block other commands, not recommended
• bgsave asynchronous background execution, does not affect other executions
• Automatic, specify rules, execute automatically

RDB related configuration files

In the example below the behaviour will be to save:
 206 #   after 900 sec (15 min) if at least 1 key changed
 207 #   after 300 sec (5 min) if at least 10 keys changed
 208 #   after 60 sec if at least 10000 keys changed
save 900 1   #在九百秒内有一个触发我就执行
save 300 10
save 60 10000
dbfilename dump.rdb
dir ./     #编泽编译安装,默认RDB文件存放在启动redis的工作目录,建议明确指定存入目录
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes

Advantages and disadvantages of RDB mode

Advantages of RDB mode

• RDB is more suitable for full backup mode to save the data at a certain point in time. You can execute the redis command bgsave (non-blocking, background execution) or save (will block the write operation, recommended) command to self-define the backup at the time point, and you can keep more A backup, when a problem occurs, it is convenient to restore to a different time node, which is very suitable for backup, and this file format supports many third-party tools for subsequent data analysis,
  such as: RDB files can be backed up every hour within the last 24 hours , And every day of every month, also back up an RDB file, in this way, even if you encounter problems, you can restore the data set to a different version at any time
• RDB can maximize the performance of Redis . When the parent process saves the RDB file, the only thing to do is to fork out a child process, and then this child process will process all the next saved work, the parent process does not need to execute any disk I/O operating
• REB recovers a lot of data, such as several G data, faster than AOF

Disadvantages of RDB

• The REB method cannot save data in real time and may lose the data from the last RDB backup to the current memory.
  If you need to try to avoid data loss when the server fails, then RDB is not for you. Although Redis allows you to set different save points to control the frequency of saving RDB files, because ROB files need to save the state of the entire data set, it is not an easy operation. So you may save the RDB file at least 5 minutes. In this case, once a downtime occurs, you may lose several minutes of data.
• Version compatible with RDB format problem. When the amount of data is very large, it takes a while to save from the parent process to the child process to the RDB file, which may be milliseconds or seconds, depending on the disk IO performance. When the data set is relatively large, fork() It may be very time-consuming, causing the server to stop processing the client within a certain period of time; if the data set is very large and the CPU time is very tight, then this stopping time may even be as long as a full second or more. Although AOF rewriting also requires fork(), no matter how long the execution interval of AOF rewriting is, there will be no loss of data durability

Practice the script for manually backing up RDB files

[root@centos7 ~]#vim /apps/redis/etc/redis.conf
save ""
dbfilename dump_6379.rdb
dir "/data/redis"
appendonly no

[root@centos8 ~]#cat redis_backup_rdb.sh
#!/bin/bash
. /etc/init.d/functions
BACKUP=/backup/redis-rdb
DIR=/data/redis
FILE=dump_6379.rdb
PASS=123456
redis-cli -h 127.0.0.1 -a $PASS --no-auth-warning bgsave
result=`redis-cli -a 123456 --no-auth-warning info Persistence |grep
rdb_bgsave_in_progress| sed -rn 's/.*:([0-9]+).*/\1/p'`
until [ $result -eq 0 ] ;do
  sleep 1
  result=`redis-cli -a 123456 --no-auth-warning info Persistence |grep
rdb_bgsave_in_progress| sed -rn 's/.*:([0-9]+).*/\1/p'`
done
DATE=`date +%F_%H-%M-%S`
[ -e $BACKUP ] || { mkdir -p $BACKUP ; chown -R redis.redis $BACKUP; }
mv $DIR/$FILE $BACKUP/dump_6379-${DATE}.rdb
action "Backup redis RDB"
#执行
[root@centos8 ~]#bash redis_backup_rdb.sh
Background saving started
Backup redis RDB                      [ OK ]
[root@centos8 ~]#ll /backup/redis-rdb/ -h
total 143M
-rw-r--r-- 1 redis redis 143M Oct 21 11:08 dump_6379-2020-10-21_11-08-47.rdb

Example: Observe the execution process of save and bgsave 67255

#阻塞
#生成临时文件

Example: Auto save

[root@centos7 ~]#vim /apps/redis/etc/redis.conf
save 60 3
#测试60s内修改3个key,验证是否生成RDB文件

AOF mode

The working mode of AOF mode (two pictures are convenient for everyone to understand)

1. Append all write commands to the aof buffer
2. The AOF buffer is synchronized to the hard disk according to the corresponding appendfsync configuration
3. Rewrite AOF files regularly
4. When Redis restarts, you can load the AOF file for data recovery

AOF: AppendOnylFile, follow the order of operations to append operations to the end of the specified log file.
AOF uses the copy-on-write mechanism like RDB. AOF defaults to fsync once per second, which means that the executed commands will be saved to the AOF file, even if redis If the server fails, only data within 1 second will be lost. You can also set different fsync strategies always, that is, set to execute fsync every time a command is executed, fsync will execute the thread in the background, so the main thread can continue to process the user’s Normal request without being affected by the I/O written to the AOF file.
Both RDB and AOF are enabled. When recovering, the default AOF file priority is higher than the RDB file, that is, the AOF file will be used for recovery.
Note!!!: The default AOF mode is Closed, after opening AOF for the first time, and restarting the service to take effect, the priority of AOF is higher than RDB, and AOF does not have files by default, which will cause all data to be lost.

AOF rewrite

• Rewrite some duplicate, mergeable, and expired data into a new AOF file, thereby saving hard disk space occupied by AOF backup and speeding up the recovery process
• You can manually execute bgrewriteaof to trigger AOF, or define an automatic rewrite strategy
  

Note!!!: The AOF mode is turned off by default. After the AOF is turned on for the first time and the service is restarted to take effect, the priority of AOF is higher than that of RDB, and AOF has no files by default, which will cause all data to be lost.

[root@centos8 ~]#ll /var/lib/redis/
total 314392
-rw-r--r-- 1 redis redis 187779391 Oct 17 14:23 dump.rdb
[root@centos8 ~]#redis-cli
127.0.0.1:6379> config get appendonly
1) "appendonly"
2) "no"
127.0.0.1:6379> config set appendonly  yes
OK
[root@centos8 ~]#ll /var/lib/redis/
total 314392
-rw-r--r-- 1 redis redis 187779391 Oct 17 14:23 dump.rdb
-rw-r--r-- 1 redis redis  85196805 Oct 17 14:45 temp-rewriteaof-2146.aof
[root@centos8 ~]#ll /var/lib/redis/
total 366760
-rw-r--r-- 1 redis redis 187779391 Oct 17 14:45 appendonly.aof
-rw-r--r-- 1 redis redis 187779391 Oct 17 14:23 dump.rdb
[root@centos8 ~]#vim /etc/redis.conf
appendonly yes #改为yes
#config set appendonly yes 可以同时看到下面显示

AOF related configuration

appendonly yes              #是否开启AOF日志记录，默认redis使用的是rdb方式持久化，这种方式在许多应用中已经足够用了，但是redis如果中途宕机，会导致可能有几分钟的数据丢失(取决于dump数据的间隔时间)，根据save来策略进行持久化，Append Only File是另一种持久化方式，可以提供更好的持久化特性，Redis会把每次写入的数据在接收后都写入 appendonly.aof 文件，每次启动时Redis都会先把这个文件的数据读入内存里，先忽略RDB文件。默认不启用此功能
appendfilename "appendonly-${port}.aof"          #文本文件AOF的文件名，存放在dir指令指定的目录中
appendfsync everysec    #aof持久化策略的配置
#no表示由操作系统保证数据同步到磁盘,Linux的默认fsync策略是30秒，最多会丢失30s的数据
#always表示每次写入都执行fsync，以保证数据同步到磁盘,安全性高,性能较差
#everysec表示每秒执行一次fsync，可能会导致丢失这1s数据,此为默认值,也生产建议值
dir /path               #快照文件保存路径，示例：dir "/apps/redis/data"

no-appendfsync-on-rewrite yes     #在aof rewrite期间,是否对aof新记录的append暂缓使用文件同步策略,主要考虑磁盘IO开支和请求阻塞时间。
#默认为no,表示"不暂缓",新的aof记录仍然会被立即同步到磁盘，是最安全的方式，不会丢失数据，但是要忍受阻塞的问题
#为yes,相当于将appendfsync设置为no，这说明并没有执行磁盘操作，只是写入了缓冲区，因此这样并不会造成阻塞（因为没有竞争磁盘），但是如果这个时候redis挂掉，就会丢失数据。丢失多少数据呢？Linux的默认fsync策略是30秒，最多会丢失30s的数据,但由于yes性能较好而且会避免出现阻塞因此比较推荐
#rewrite 即对aof文件进行整理,将空闲空间回收,从而可以减少恢复数据时间
auto-aof-rewrite-percentage 100             #当Aof log增长超过指定百分比例时，重写AOF文件，设置为0表示不自动重写Aof日志，重写是为了使aof体积保持最小，但是还可以确保保存最完整的数据
auto-aof-rewrite-min-size 64mb              #触发aof rewrite的最小文件大小
aof-load-truncated yes                      #是否加载由于某些原因导致的末尾异常的AOF文件(主进程被kill/断电等)，建议yes
aof-use-rdb-preamble no #redis4.0新增RDB-AOF混合持久化格式，在开启了这个功能之后，AOF重写产生的文件将同时包含RDB格式的内容和AOF格式的内容，其中RDB格式的内容用于记录已有的数据，而AOF格式的内容则用于记录最近发生了变化的数据，这样Redis就可以同时兼有RDB持久化和AOF持久化的优点（既能够快速地生成重写文件，也能够在出现问题时，快速地载入数据）,默认为no,即不启用此功能

Example: dynamically modify the configuration and automatically generate appendonly.aof file

127.0.0.1:6379> CONFIG set appendonly yes

Manually execute AOF to rewrite BGREWRITEAOF command

• BGREWRITEAOF
  time complexity: O(N), N is the number of data to be appended to the AOF file.
  Perform an AOF file rewrite operation. Rewriting will create an optimized version of the current AOF file.
• Even if BGREWRITEAOF fails, there will be no data loss, because the old AOF file will not be modified before BGREWRITEAOF succeeds.
• The rewrite operation will only be triggered when no other persistence work is executed in the background, that is to say:
  if the child process of Redis is performing the snapshot saving work, then the AOF rewrite operation will be scheduled (scheduled) and wait until the work is saved Perform AOF rewriting after completion. In this case, the return value of BGREWRITEAOF is still OK, but an additional message will be added, stating that BGREWRITEAOF cannot be executed until the save operation is completed. In Redis 2.6 or above, you can use the INFO [section] command to check whether BGREWRITEAOF is booked.
• If other AOF file rewriting is already being executed, then BGREWRITEAOF returns an error, and this new
  BGREWRITEAOF request will not be scheduled for the next execution.
• Starting with Redis 2.4, AOF rewriting is triggered by Redis itself, and BGREWRITEAOF is only used to manually trigger the rewriting operation.

Example: manual bgrewriteaof

127.0.0.1:6379> BGREWRITEAOF
Background append only file rewriting started

#同时可以观察到下面显示

Advantages and disadvantages of AOF mode

Advantages of AOF mode

• Data security is relatively high. According to the used fsync strategy (fsync is to synchronize all the modified files of redis in memory to the storage device), the default is appendfsync everysec, that is, fsync is executed once per second. In this configuration, Redis still Good performance can be maintained, and even if a failure occurs, only one second of data will be lost (fsync will be executed in a background thread, so the main thread can continue to work hard to process command requests)
• Because this mechanism uses append mode for the write operation of the log file, there is no need to seek during the writing process, and even if there is a downtime, it will not destroy the existing content in the log file. However, if this operation only writes half of the data and the system crashes, don’t worry, you can use the redis-check-aof tool to solve the data consistency problem before Redis is started next time
• Redis can automatically rewrite AOF in the background when the volume of the AOF file becomes too large. The new AOF file after rewriting contains the minimum set of commands required to restore the current data set. The entire rewriting operation is absolutely safe, because Redis is in the process of creating a new AOF file, the append mode continuously appends the modified data to the existing AOF file, even if the rewriting process stops, the existing AOF file is still Will not be lost. Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending the new AOF file.
• AOF contains a clearly formatted, easy-to-understand log file to record all modification operations. In fact, the data can also be reconstructed through this file. The AOF file stores all the write operations performed on the database in an orderly manner. These write operations are saved in the format of the Redis protocol, so the content of the AOF file is very easy to be read by people. , Parse the file is also very easy. Exporting AOF files is also very simple:
• For example, if you accidentally execute the FLUSHALL. command, but as long as the AOF file is not rewritten, just stop the server, remove the FLUSHAL command at the end of the AOF file, and restart Redis, and you can restore the data set to
  FLUSHALL. The previous state.

Disadvantages of AOF mode

• Even if some operations are repeated, they will all be recorded. The file size of AOF is larger than that of RDB format.
• AOF is slower when recovering large data sets than RDB
• Depending on the fsync strategy, AOF may be slower than RDB
• More possibilities for bugs

Choice of RDB and AOF

• If it is mainly used as a caching function, or can withstand the loss of several minutes of data, usually the production environment generally only needs to enable RDB, which is also the default value
• If the data needs to be stored persistently and cannot be lost at all, you can choose to turn on RDB and AOF at the same time. Generally, it is not recommended to only turn on AOF