[Redis study notes] four, Redis persistence operations

Redis persistence

RDB(Redis DataBase)

principle

Within a specified time interval memory snapshot of a dataset written to disk, it is to restore the snapshot files directly read memory.

Principle: Redis creates a separate (fork) a child process to persist, will be written to a temporary file in the data first, to be persistent processes are received, replace the last good persistence file and then this temporary file . Throughout the process, the main process is not carried out any IO operations , which ensures a high level of performance.

fork a copy of the job is the same process with the current process. All data (variables, environment variables, program counter, etc.) and the value of the new process are consistent with the original process, but is a new process, and as a child process of the original process.

If you need to respond to large-scale data, and not so sensitive to the integrity of the data recovery, then RDB ways to be more efficient.

** RDB drawback: the last data persistence may lose the book, can not be written to the hard disk. ** Because RDB rule is when there is an update operation many times how long the trigger when data synchronization. If just one data synchronization, this time Redis still write, but the condition does not meet the trigger data synchronization, this time Redis service crashes, then the data is written to the file can not be synchronized.

rdb is dump.rdb default save files, can be configured by Redis.conf

How to trigger a snapshot of RDB

In order to facilitate, for example, will save the rule change:

save 900 1
save 300 10
## 更改为20秒内5次写入操作即保存
save 20 5

Example:

## redis_conf 默认dir为redis服务的启动目录 ./;现在可以发现这目录是空的
test_redis ls
## 在test_redis下启动redis服务
➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
Password:
➜  test_redis redis-cli -h 127.0.0.1 -p 6379  -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> set k2 v2
OK
127.0.0.1:6379> set k3
(error) ERR wrong number of arguments for 'set' command
127.0.0.1:6379> set k3 v3
OK
127.0.0.1:6379> set k4 v4
OK
127.0.0.1:6379> set k5 v5
OK

5 times a writing operation is performed in the 20s, found file generated dump.rdb

Redis.log is configured Redis startup log file

It may also be performed immediately by a data synchronization command

• save

  Just save save time, regardless of other, block the execution of other commands. If you save the implementation process is longer, then the other clients after the save command executes other commands, then the other will be blocked to save the command is executed.

  For example:

  ## 这里再进行一次 写入操作 时间点10:23 这里应该不符合任何配置的save条件，不会同步数据
  127.0.0.1:6379> set k6 v6
  OK
  ## 手动保存
  127.0.0.1:6379> save
  OK
  ## 查看最后一次快照执行成功的时间
  127.0.0.1:6379> LASTSAVE
  (integer) 1580696630 ## 转换过来是Mon Feb 03 2020 10:23:50 GMT+0800 (中国标准时间)
  

Observe the changes dump.rdb file:

​

We found immediately execute a data synchronization.

• bgsave

  Redis will operate simultaneously snapshot, the snapshot is performed asynchronously in the background, Redis can also respond to client requests. Time can get a snapshot of the last successful execution of the command by lastsave

  For example there is no longer

• flushall、flushdb

  These two commands are clear all the libraries and the current library. Execute these commands, will be immediately snapshot, but because these commands to clear the operation, the snapshot file is empty.

  

  After you can see the implementation of the flushall, even if carried out immediately snapshot, but the snapshot file is empty, and then restart the service also load than the data.

How to recover data

There the backup file directory just run Redis-server services, data can be loaded into the file when you start Redis service.

config get dir - get Redis configuration directory

As I start the Redis service at test_Redis,

➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
➜  test_redis redis-cli -h 127.0.0.1   -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> set k6 v6
OK
127.0.0.1:6379> set k7 v7
OK
127.0.0.1:6379> set k8 v8
OK
127.0.0.1:6379> set k9 v9
OK
127.0.0.1:6379> set k10 v10
OK
127.0.0.1:6379> LASTSAVE
(integer) 1580698473
## 停止服务
127.0.0.1:6379> SHUTDOWN
not connected> exit
➜  test_redis clear
## 重新启动服务
➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
➜  test_redis redis-cli -h 127.0.0.1   -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
## 加载到这些文件了
127.0.0.1:6379> keys *
1) "k9"
2) "k10"
3) "k8"
4) "k7"
5) "k6"
127.0.0.1:6379>

If I first started under test_Redis directory service and a write operation, the file will be generated in the directory. If the second when I started in another directory service, can be loaded into operation just written, to see whether there are other directories dump.rdb and whether we file that under test_Redis might otherwise start the service load not see any data or other data is loaded dump.rdb file

Disable RDB Save Rules

1. Save the configuration file in Redis.conf ""
2. Execute commands in Redis-cli: config set save ""

to sum up

advantage:

Suitable for large-scale data recovery; data integrity and consistency do not ask

Disadvantages:

A certain time interval to do a backup, so if Redis accidentally fall down, then it

All modifications will be lost after the last snapshot.

RDB often need to fork child process to save the data set to the hard disk, when the data set is relatively large when, fork process is time consuming, may lead in some Redis millisecond can not respond to client requests. fork, when the data in memory will be a clone, steep memory overhead to consider.

Note: If a file is damaged rdb, may be used redis-check-rdb to repair, for example:

➜  bin redis-check-rdb  ~/test_redis/dump.rdb
[offset 0] Checking RDB file /Users/wojiushiwo/test_redis/dump.rdb
[offset 26] AUX FIELD redis-ver = '5.0.0'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1580699961'
[offset 67] AUX FIELD used-mem = '1038144'
[offset 83] AUX FIELD aof-preamble = '0'
[offset 92] Checksum OK
[offset 92] \o/ RDB looks OK! \o/
[info] 0 keys read
[info] 0 expires
[info] 0 already expired

AOF(Append Only File)

principle

In the form of log records to each write operation, the write Redis executed all instructions recorded (read is not recorded). Only append files can not overwrite the file, starting at the beginning of Redis reads the file to reconstruct the data. In other words, Redis, then restart it based on the contents of the log file will execute the write command from the front backwards once to complete the recovery of the data.

默认AOF保存的是appendonly.aof文件，可在Redis.conf中配置。

使用

启动AOF配置

在Redis.conf中 修改默认的appendonly no，改为yes

恢复数据

将有数据的aof文件复制一份保存到对应目录(Redis.conf中配置的dir的目录，如果配置的是./ 则将aof文件复制到启动Redis服务的目录下)，重启Redis即可加载到aof文件数据

举例：

## 启动服务，会立刻生成一个文件appendonly.aof
➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
Password:
➜  test_redis redis-cli -h 127.0.0.1   -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> keys *
(empty list or set)
## 默认aof策略是每秒备份一次 所以下面的写操作会被备份到aof文件中
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> set k2 v2
OK
## 停止服务
127.0.0.1:6379> SHUTDOWN
not connected> exit
## 重启服务
➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
➜  test_redis redis-cli -h 127.0.0.1   -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
## 从aof文件中加载到数据
127.0.0.1:6379> keys *
1) "k1"
2) "k2"
127.0.0.1:6379>

查看下aof文件的内存：

*2
$6
SELECT
$1
0
*3
$3
set
$2
k1
$2
v1
*3
$3
set
$2
k2
$2
v2

可以看到确实是以日志模式记录下写操作。

但是如果aof文件被写坏了，我们看还能不能正常的恢复数据:

在aof文件中写入随机内容，如下

*2
$6
SELECT
$1
0
*3
$3
set
$2
k1
$2
v1
*3
$3
set
$2
k2
$2
v2
sdbjakdbsabdsakdbsajkk

重启服务

➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
Password:
➜  test_redis ps -ef |grep redis
  501 21743 16955   0 11:38上午 ttys001    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn redis

发现服务并没有起来，查看Redis.log日志，日志中有如下提示：

21734:M 03 Feb 2020 11:38:33.453 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>

意思是说我们的aof文件已经损坏，需要修复。

因此得出结论,aof文件被写坏了不仅不能恢复数据，连服务都起不来

接下来，修复aof文件，修复语句：redis-check-aof --fix 目录下的appendonly.aof

示例：

## 修复aof文件
➜  test_redis sudo /usr/local/bin/redis-check-aof --fix appendonly.aof
0x              51: Expected prefix '*', got: 's'
AOF analyzed: size=105, ok_up_to=81, diff=24
This will shrink the AOF from 105 bytes, with 24 bytes, to 81 bytes
Continue? [y/N]: y
## 修复成功
Successfully truncated AOF
➜  test_redis sudo /usr/local/bin/redis-server ~/redis/redis_bak.conf
➜  test_redis redis-cli -h 127.0.0.1   -p 6379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
## 数据被再次加载进来
127.0.0.1:6379> keys *
1) "k2"
2) "k1"
127.0.0.1:6379>

再次查看aof文件内容：

*2
$6
SELECT
$1
0
*3
$3
set
$2
k1
$2
v1
*3
$3
set
$2
k2
$2
v2

发现刚才那些随机内容已经被删除了。

rewrite

AOF采用文件追加的方式，文件会越来越大。为了避免这种情况，新增了重写机制。当AOF文件的大小超过设定的阈值时，reids就会启动AOF文件的内容压缩，只保留可以恢复数据的最小指令集。也可以使用命令来完成：bgrewriteaof

原理：

AOF文件持续增长而过大时，会fork出一条新进程将文件重写(先写进一个临时文件，待写完后将临时文件更名为appendonly.aof)，遍历新进程的内存中的数据，每条记录有一条set语句。重写aof文件的操作并没有读取旧的aof文件，而是将整个内存中的数据库内容用命令的方式重写到了一个新的aof文件中。

触发机制：

Redis.conf中配置的auto-aof-rewrite-min-size、auto-aof-rewrite-percentage即是触发的条件。

Redis会记录上次重写的AOF文件大小，默认配置是当AOF文件大小是上次rewrite后大小的一倍且文件大于64M时触发

总结

AOF文件是一个只进行追加的日志文件，Redis可以在aof文件体积变得过大时，自动地在后台对AOF进行重写。

AOF文件有序地保存了对数据库执行的所有写入操作，这些写入操作已Redis协议的格式保存，因此AOF文件的内容很容易被人读懂。

对于相同的数据集来说，AOF文件的体积通常要大于RDB文件的体积，根据所使用的同步策略，AOF的恢复速度和运行效率会慢于RDB。AOF的appendfsync everysec策略效率较好，appendfsync no策略效率和RDB相当。

Redis的持久化总结

RDB持久化方式能够在指定的时间间隔能对你的数据进行快照存储。

AOF持久化方式记录每次对服务器写的操作,当服务器重启的时候会重新执行这些命令来恢复原始的数据,AOF命令以Redis协议追加保存每次写的操作到文件末尾。Redis还能对AOF文件进行后台重写,使得AOF文件的体积不至于过大。

同时开启两种持久化方式

在这种情况下,当Redis重启的时候会优先载入AOF文件来恢复原始的数据,因为在通常情况下AOF文件保存的数据集要比RDB文件保存的数据集要完整。RDB的数据不实时，同时使用两者时服务器重启也只会找AOF文件。

由于AOF的特点是文件在不断变化不好备份，因此RDB更适合用于备份数据库。

建议只在slave上持久化RDB文件，设置15分钟备份一次即可

如果配置中也开启了AOF,优势是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只load自己的AOF文件就可以了。代价：

1. 带来了持续的IO(服务重启时会加载aof文件，相当于将aof文件中的命令重新执行了一次)。
2. AOF will rewtire new data generated by the new file is written obstruction caused you almost inevitable. Should therefore be minimized in the frequency AOF rewrite, AOF rewritten based 64M size is too small default value (frequent may exceed this threshold for rewrite). May be rewritten to a suitable default value exceeds 100% of its original size.

If you do not open the AOF, rely on the master-slave assignment can achieve high availability, can save a lot of IO is also reduced when the system is brought rewrite fluctuations. However, if the master and slave have collapsed, then lose data over time (to compare the latest in a primary selection from rdb file restart).