Redis persistence mechanism RDB and AOF

1 Endurance RDB

Redis DataBase

1.1 RDB concept

At regular intervals, the data in the memory is written into a temporary file on the disk as a snapshot, and the snapshot file is read into the memory when restoring. If the machine is restarted, the data in the memory will definitely be gone, so after restarting redis, it will be restored.

1.2 Backup and restore

Memory backup --> Temporary disk files
Temporary file --> restore to memory

1.3 Advantages and disadvantages of RDB

1.3.1 Advantages

Backup every once in a while, full backup
Disaster recovery is simple and can be transmitted remotely
When the child process is backed up, the main process will not have any io operations (there will be no write modification or deletion) to ensure the integrity of the backup data
Compared with AOF, it can be quickly restarted and restored when there are larger files

1.3.2 Disadvantages

In the event of a failure, the last backup data may be lost
The ratio of memory occupied by the child process will be exactly the same as that of the parent process, which will cause a burden on the CPU
Since the scheduled full backup is a heavyweight operation, it cannot be processed for real-time backup.

1.4 RDB configuration

save 900 1
save 300 10
save 60 10000
save 10 3

* 如果1个缓存更新，则15分钟后备份
* 如果10个缓存更新，则5分钟后备份
* 如果10000个缓存更新，则1分钟后备份
* 演示：更新3个缓存，10秒后备份
* 演示：备份dump.rdb，删除重启

1 stop-writes-on-bgsave-error

yes: If an error occurs during the save process, the write operation is stopped
no: may cause data inconsistency

2 rdbcompression

yes: enable rdb compression mode
no: Closed, will save cpu loss, but the file will be large, the same reason as nginx

3 rdbchecksum

yes: Use CRC64 algorithm to check data for RDB, there is a 10% performance loss
no: no verification

1.5 RDB test

1.5.1 Modify the configuration file

[root@localhost ~]# cd /usr/local/redis
[root@localhost redis]# vi 6379.conf

#   It is also possible to remove all the previously configured save
#   points by adding a save directive with a single empty string argument
#   like in the following example:
#
#   save ""

save 900 1
save 300 10
save 60 10000
save 10 3

root@localhost redis]# /etc/init.d/redis_init_script stop
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Waiting for Redis to shutdown ...
Redis stopped
[root@localhost redis]# /etc/init.d/redis_init_script start
Starting Redis server...
16429:C 08 Feb 2021 17:04:01.087 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16429:C 08 Feb 2021 17:04:01.094 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=16429, just started
16429:C 08 Feb 2021 17:04:01.094 # Configuration loaded
[root@localhost redis]# cd /usr/local/redis/working
[root@localhost working]# ll --full-time
总用量 8
-rw-r--r--. 1 root root 5713 2021-02-08 17:03:56.993740635 +0800 dump.rdb

1.5.2 Store data

[root@localhost ~]# redis-cli
127.0.0.1:6379> auth auskat
OK
127.0.0.1:6379> set aa aa
OK
127.0.0.1:6379> set bb bb
OK
127.0.0.1:6379> set cc ccc
OK

1.5.3 View results

/usr/local/redis/workingThis path is the configuration file dirpath configuration

dir /usr/local/redis/working

[root@localhost working]# ll --full-time
总用量 8
-rw-r--r--. 1 root root 5740 2021-02-08 17:09:09.390347124 +0800 dump.rdb

2 Persistent AOF

Append Only File

RDB will lose the last backed up rdb file. It is okay if the overall integrity of the data is not high, but if the integrity of the data is pursued, then AOF should be considered.

2.1 AOF concept

The write operation requested by the user is recorded in the form of a log. Read operations will not be recorded, because write operations will be stored.
The file is appended instead of modified
Redis's aof recovery is actually to read and write the appended file from the beginning to the end.

2.2 Advantages and disadvantages of AOF

2.2.1 Advantages

AOF is more durable and can be backed up in seconds. If a problem occurs, only the last second of data will be lost, which greatly increases reliability and data integrity. So AOF can be backed up once per second, using fsync operation.
It is appended in the form of log. If the disk is full, the redis-check-aof tool will be executed
When the data is too large, redis can automatically rewrite AOF in the background. When redis continues to append the log to the old file, rewriting is also very safe and will not affect the client's read and write operations.
All write operations contained in the AOF log will make it easier for redis to analyze and recover

2.2.2 Disadvantages

The same data, the same data, AOF is larger than RDB
For different synchronization mechanisms, AOF will be slower than RDB, because AOF will back up for write operations every second, which is slightly lower than RDB. Backing up fsync every second is no problem, but if the client does a backup fsync every time it writes, then the performance of redis will decrease.
AOF has had a bug, that is, the data is incomplete when the data is restored. This makes AOF more fragile and prone to bugs, because AOF is not as simple as RDB. In order to prevent BUG generation, AOF will not be reconstructed according to the old instructions. Refactoring is based on the data instructions existing in the cache at the time, so that it is more robust and reliable

2.3 AOF configuration

# AOF 默认关闭，yes可以开启
appendonly no
# AOF 的文件名
appendfilename "appendonly.aof"
# no：不同步
# everysec：每秒备份，推荐使用
# always：每次操作都会备份，安全并且数据完整，但是慢性能差
appendfsync everysec
# 重写的时候是否要同步，no可以保证数据安全
no-appendfsync-on-rewrite no
# 重写机制：避免文件越来越大，自动优化压缩指令，会fork一个新的进程去完成重写动作，新进程里的内存数据会被重写，此时旧的AOF文件不会被读取使用，类似RDB
# 当前AOF文件的大小是上次AOF大小的100% 并且文件体积达到64m，满足两者则触发重写
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

2.4 AOF test

2.4.1 Modify the configuration file

# AOF 默认关闭，yes可以开启
appendonly yes

root@localhost redis]# /etc/init.d/redis_init_script stop
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Waiting for Redis to shutdown ...
Redis stopped
[root@localhost redis]# /etc/init.d/redis_init_script start
Starting Redis server...
16429:C 08 Feb 2021 17:04:01.087 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16429:C 08 Feb 2021 17:04:01.094 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=16429, just started
16429:C 08 Feb 2021 17:04:01.094 # Configuration loaded
[root@localhost redis]# cd /usr/local/redis/working
[root@localhost working]# ll --full-time
总用量 8
-rw-r--r--. 1 root root    0 2021-02-08 17:15:23.437467996 +0800 appendonly.aof
-rw-r--r--. 1 root root 5740 2021-02-08 17:15:16.547410508 +0800 dump.rdb

2.4.2 Store data

[root@localhost ~]# redis-cli
127.0.0.1:6379> auth auskat
OK
127.0.0.1:6379> set aaa aa
OK
127.0.0.1:6379> set bbb bb
OK
127.0.0.1:6379> set ccc ccc
OK

2.4.3 View results

[root@localhost working]# ll --full-time
总用量 8
-rw-r--r--. 1 root root 116 2021-02-08 17:16:34.004056771 +0800 appendonly.aof
-rw-r--r--. 1 root root 124 2021-02-08 17:16:34.177058213 +0800 dump.rdb
[root@localhost working]# vi appendonly.aof

*2
$6
SELECT
$1
0
*3
$3
set
$3
aaa
$3
aaa
*3
$3
set
$3
bbb
$3
bbb
*3
$3
set
$3
ccc
$3
ccc

2.4.4 Data Recovery

[root@localhost ~]# redis-cli
127.0.0.1:6379> auth auskat
OK
127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> keys *
(empty list or set)

修改AOF文件删除最后一行命令，flushdb，然后重启redis服务

[root@localhost working]# vi appendonly.aof

[root@localhost working]# /etc/init.d/redis_init_script stop
Stopping ...
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
Waiting for Redis to shutdown ...
Redis stopped
[root@localhost working]# /etc/init.d/redis_init_script start
Starting Redis server...
16595:C 08 Feb 2021 17:19:51.643 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16595:C 08 Feb 2021 17:19:51.643 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=16595, just started
16595:C 08 Feb 2021 17:19:51.643 # Configuration loaded

[root@localhost ~]# redis-cli
127.0.0.1:6379> auth auskat
OK
127.0.0.1:6379> keys *
1) "bbb"
2) "ccc"
3) "aaa"

3 Persistence scheme

If you can accept a period of cache loss, you can use RDB
If the data integrity requirements are relatively high, then use AOF
Under normal circumstances, the method that will be used is to use RDB and AOF together, RDB is used for cold backup, and different versions can be restored at different times, and AOF is used for hot backup.

Ensure that the data is only lost for 1 second. When AOF is damaged and unavailable, RDB recovery is used. If the two are combined, Redis recovery will load AOF first, and if there is a problem with AOF, load RDB to achieve the purpose of both hot and cold.

4 Related information

The blog post is not easy, everyone who has worked so hard to pay attention and praise, thank you