[FIG movable demonstrates] the Redis persistence RDB / AOF Detailed Practice

Redis is an open source (BSD license), the data structure stored in the memory system that can be used as a database, cache and messaging middleware. It is very rich supported data types, such as a string, chain, set, and hashing the like, and also supports a variety of sorting.

What is persistence?

Sentence can be persistent summarized as follows: to save the data (e.g., memory objects) to be permanently stored in the memory device. Persistence is the major application in the database, or disk file, XML data files, and so on objects stored in the storage memory.

From the application layer and the system layer appreciated persistence

At the same time, also from the application layer and system layer two levels to understand the persistence:

Application layer: If you close ( Close ) your application and restart the previous data is still there.

System layer: If you close ( Shutdown ) your system (computer) and then restart the previous data is still there.

Why Redis persistence?

Redis data types are supported in the push / pop, add / remove and on the intersection and union, and difference richer operation, and these operations are atomic. On this basis, Redis supports a variety of different ways of sorting. Like with Memcached, in order to ensure efficiency, the data is cached in memory.

Yes, the data is cached in memory, when you shut down the system or reboot the system, the data is cached in memory will disappear, never regained. Therefore, in order to make long-term preservation of data, it is necessary to put Redis data cache do persistent storage.

Redis how to achieve persistence?

Early in the design, Redis has been taken into account this problem. Official offers a variety of different levels of data persistence ways:

1, RDB persistent manner can the specified time interval can store a snapshot of your data.

2, AOF persistent recorded each write operation to the server when the server restart will re-execute these commands to restore the original data, AOF command to save an additional protocol redis each write operation to the end of the file can .Redis AOF overwriting files on the background, so that the volume is too large will not AOF file.

3, if you want your data exists only when the server is running, you can not use any persistent way.

4, you can also open two kinds of persistence simultaneously, in this case, when redis restart priority load AOF file to restore the original data, as stored under normal circumstances AOF file data set than RDB file data sets to complete.

If you do not know which way a persistent level of the selection, then we start to look at ways AOF and RDB way, what kind of difference, and what their respective advantages and disadvantages, after learning finished, again consider the choice which species level.

Comparative advantage RDB way AOF way

First, let's look at the advantages of two ways for the official description and make a comparison, then take a look at the shortcomings described in two ways.

RDB advantage of the way

RDB is a very compact file, which holds a certain point in time data collection, ideal for backup data sets, such as what you can save data in the past 24 hours in each of the Times, while preserving the past 30 days every day data, so that even if a problem you can restore to a different version of the data set according to demand.
RDB is a compact, single file, easily transferred to another remote data center, ideal for disaster recovery.
RDB RDB file when you save the only parent needs to do is fork a child process, then all the work done by the child, the parent is not required to do other IO operations, so RDB persistent way to maximize the performance of Redis .
Compared with the AOF, when restoring large data sets, RDB way faster.

When Redis need to save dump.rdb a file, the server does the following:

Redis call forks. Have both parent and child processes.
The child process data set is written to a temporary file RDB.
When the child process to complete the writing of a new RDB file, Redis replace the original file with the new RDB RDB file, and delete the old file RDB.

This way of working makes Redis can be copied from the write ( copy-on-write) to benefit mechanism.

AOF advantage of the way

AOF will make your use more durable Redis:

你可以使用不同的 fsync 策略：无 fsync、每秒 fsync 、每次写的时候 fsync .使用默认的每秒 fsync 策略, Redis 的性能依然很好( fsync 是由后台线程进行处理的,主线程会尽力处理客户端请求),一旦出现故障，你最多丢失1秒的数据。
AOF文件是一个只进行追加的日志文件,所以不需要写入seek,即使由于某些原因(磁盘空间已满，写的过程中宕机等等)未执行完整的写入命令,你也也可使用redis-check-aof工具修复这些问题。
Redis 可以在 AOF 文件体积变得过大时，自动地在后台对 AOF 进行重写：重写后的新 AOF 文件包含了恢复当前数据集所需的最小命令集合。整个重写操作是绝对安全的，因为 Redis 在创建新 AOF 文件的过程中，会继续将命令追加到现有的 AOF 文件里面，即使重写过程中发生停机，现有的 AOF 文件也不会丢失。而一旦新 AOF 文件创建完毕，Redis 就会从旧 AOF 文件切换到新 AOF 文件，并开始对新 AOF 文件进行追加操作。
AOF 文件有序地保存了对数据库执行的所有写入操作，这些写入操作以 Redis 协议的格式保存，因此 AOF 文件的内容非常容易被人读懂，对文件进行分析（parse）也很轻松。导出（export） AOF 文件也非常简单：举个例子，如果你不小心执行了 FLUSHALL 命令，但只要 AOF 文件未被重写，那么只要停止服务器，移除 AOF 文件末尾的 FLUSHALL 命令，并重启 Redis ，就可以将数据集恢复到 FLUSHALL 执行之前的状态。

优点对比总结

RDB 方式可以保存过去一段时间内的数据，并且保存结果是一个单一的文件，可以将文件备份到其他服务器，并且在回复大量数据的时候，RDB 方式的速度会比 AOF 方式的回复速度要快。

AOF 方式默认每秒钟备份1次，频率很高，它的操作方式是以追加的方式记录日志而不是数据，并且它的重写过程是按顺序进行追加，所以它的文件内容非常容易读懂。可以在某些需要的时候打开 AOF 文件对其编辑，增加或删除某些记录，最后再执行恢复操作。

RDB 方式与 AOF 方式的缺点对比

RDB 方式的缺点

如果你希望在 Redis 意外停止工作（例如电源中断）的情况下丢失的数据最少的话，那么 RDB 不适合你.虽然你可以配置不同的save时间点(例如每隔 5 分钟并且对数据集有 100 个写的操作),是 Redis 要完整的保存整个数据集是一个比较繁重的工作,你通常会每隔5分钟或者更久做一次完整的保存,万一在 Redis 意外宕机,你可能会丢失几分钟的数据。
RDB 需要经常 fork 子进程来保存数据集到硬盘上,当数据集比较大的时候, fork 的过程是非常耗时的,可能会导致 Redis 在一些毫秒级内不能响应客户端的请求。如果数据集巨大并且 CPU 性能不是很好的情况下,这种情况会持续1秒, AOF 也需要 fork ,但是你可以调节重写日志文件的频率来提高数据集的耐久度。

AOF 方式的缺点

对于相同的数据集来说，AOF 文件的体积通常要大于 RDB 文件的体积。
根据所使用的 fsync 策略，AOF 的速度可能会慢于 RDB 。在一般情况下，每秒 fsync 的性能依然非常高，而关闭 fsync 可以让 AOF 的速度和 RDB 一样快，即使在高负荷之下也是如此。不过在处理巨大的写入载入时，RDB 可以提供更有保证的最大延迟时间（latency）。

缺点对比总结

RDB 由于备份频率不高，所以在回复数据的时候有可能丢失一小段时间的数据，而且在数据集比较大的时候有可能对毫秒级的请求产生影响。

AOF 的文件提及比较大，而且由于保存频率很高，所以整体的速度会比 RDB 慢一些，但是性能依旧很高。

工作原理

AOF rewrite and RDB create a snapshot, as are the clever use of copy-on-write mechanism:

Redis execute fork (), and now have both parent and child processes.
The contents of the new child process began AOF file written to a temporary file.
For all new write command execution, the parent process while they accumulate into a memory cache, while these changes will append to the existing file AOF, even in this kind of rewriting halfway down the existing file AOF still safe.
When the child process is complete rewrite work, it sends a signal to the parent process, the parent process after receiving the signal, all of the additional data in cache memory to the end of the new AOF file.
Now Redis atomically replace the old file with a new file, after all of the commands will be added directly to the end of the new AOF file.

Put into practice to achieve, RDB and the AOF

RDB way persistent and open configuration

Redis persistence mode is the default RDB, and is enabled by default. There are ways to save RDB divided into active and passive preservation preservation. Save the initiative can be entered in redis-cli in save to; passive preservation need to meet the trigger condition set in the configuration file, the current official default trigger conditions can be redis.conf seen in:

save 900 1
save 300 10
save 60 10000
复制代码

Its meaning is:

服务器在900秒之内，对数据库进行了至少1次修改
服务器在300秒之内，对数据库进行了至少10次修改。
服务器在60秒之内，对数据库进行了至少10000次修改。
复制代码

After the trigger condition is met, the data will be saved as a snapshot, precisely because of this it says RDB data integrity is smaller than the AOF.

After triggering storage conditions, will generate a specified directory named in dump.rdb the file, wait until the next time you start Redis, Redis will go to read in the directory dump.rdb file, which will restore the data to the Redis.

Where is it in this directory?

We can enter commands in the client config get dirview:

gannicus@$ src/redis-cli
127.0.0.1:6379> config get dir
1) "dir"
2) "/home/gannicus/Documents/redis-5.0.0"
127.0.0.1:6379> 
复制代码

Return results "/home/gannicus/Documents/redis-5.0.0"that store dump.rdb directory.

Redis Imprint

Before the test, explain the premise. redis Archive directly from the official website to download, unpack the redis-x.x.x folder, such as I was redis-5.0.0, and then enter the folder, redis-5.0.0 use the project root directory makecommand to install.

RDB passive trigger to save the test

Just mention it divided into active and passive trigger to save, and now we have to test passive trigger. First start redis-server, and then open the client redis-cli , to add a few records:

127.0.0.1:6379> set lca 1
OK
127.0.0.1:6379> set lcb 1
OK
127.0.0.1:6379> set lcc 1
OK
127.0.0.1:6379> set lcd 1
OK
127.0.0.1:6379> set lce 1
OK
127.0.0.1:6379> set lcf 1
OK
127.0.0.1:6379> set lcg 1
OK
127.0.0.1:6379> set lch 1
OK
127.0.0.1:6379> set lci 1
OK
127.0.0.1:6379> set lcj 1
OK
127.0.0.1:6379> set lck 1
OK
127.0.0.1:6379> set lcl 1
OK
127.0.0.1:6379> set lcm 1
OK
复制代码

We can see, adding a total of 13 records:

127.0.0.1:6379> keys *
 1) "lca"
 2) "lcd"
 3) "lcg"
 4) "lce"
 5) "lcb"
 6) "lcm"
 7) "lcf"
 8) "lci"
 9) "lcl"
10) "lcc"
11) "lck"
12) "lcj"
13) "lch"
127.0.0.1:6379> 
复制代码

And then found redis-serverthe end of the log window appear the following tips:

21971:M 21 Oct 2018 16:52:44.062 * 10 changes in 300 seconds. Saving...
21971:M 21 Oct 2018 16:52:44.063 * Background saving started by pid 22552
22552:C 21 Oct 2018 16:52:44.066 * DB saved on disk
21971:M 21 Oct 2018 16:52:44.165 * Background saving terminated with success
复制代码

From English prompt may read about the content, it detects that the recording has been altered 10 300 seconds, we just add the data records 13, satisfies redis.conf the conditions for data stored in the RDB, the data holding operations performed here, and Tip out the process opened up a 22,552 save operation, and finally prompted to save successfully.

And see within the directory has dump.rdb files generated.

Now the redis process kill, what data will be saved?

Command kill -9 pid (pid is the process ID) simulation Redis close unexpectedly, and then start Redis, we take a look, in the end is only saved 10 or 13 records all preserved?

127.0.0.1:6379> keys *
 1) "lcb"
 2) "lcj"
 3) "lcd"
 4) "lch"
 5) "lci"
 6) "lcc"
 7) "lcf"
 8) "lce"
 9) "lca"
10) "lcg"
127.0.0.1:6379> 
复制代码

View restart records and found 13 records in only 10 records will be saved, which also confirms the previously mentioned, data integrity RDB approach is unreliable, unless broken the moment happens to be trigger conditions are met under number.

Close RDB

Just mentioned, it is enabled by default, if you do not need it can configure these three commented out in the configuration file, and add save ""to:

  save ""
# save 900 1
# save 300 10
# save 60 10000
复制代码

Needs to be restarted after saving the configuration file Redis service to take effect, then continue to add more than a dozen records:

127.0.0.1:6379> keys *
 1) "lcb"
...
23) "lca"
24) "lcg"
127.0.0.1:6379> 
复制代码

On the basis of the existing 10 before I go any further 14 records, the same through killto simulate Redis close unexpectedly, and then start the service to see whether the data is also stored:

127.0.0.1:6379> keys *
 1) "lcb"
 2) "lcj"
 3) "lcd"
 4) "lch"
 5) "lci"
 6) "lcc"
 7) "lcf"
 8) "lce"
 9) "lca"
10) "lcg"
127.0.0.1:6379> 
复制代码

It found 14 records added back has not been saved, the time to recover data just restored 10 before. And observe the Redis server log window, did not find as before triggering saving tips, proven way RDB has been closed.

RDB initiative to save the test

By configuring the file is closed passive trigger, then take the initiative to shut down if they would enter into force it?

In Redis client (redis-cli) by deldeleting several records command, and then enter the savecommand to execute the save operation:

127.0.0.1:6379> keys *
 1) "lcc"
 2) "lch"
 3) "lcb"
 4) "lci"
 5) "lce"
 6) "lcj"
 7) "lcg"
 8) "lca"
 9) "lcd"
10) "lcf"
127.0.0.1:6379> del lca lcb lcc
(integer) 3
127.0.0.1:6379> save
OK
127.0.0.1:6379> 
复制代码

You can see redis-serverthe new log Hint: 22598:M 21 Oct 2018 17:22:31.365 * DB saved on diskit tells us that the data has been saved.

So continue to simulate abnormal closed, then open the service to see if really saved these actions:

127.0.0.1:6379> keys *
1) "lci"
2) "lcj"
3) "lcd"
4) "lcg"
5) "lcf"
6) "lce"
7) "lch"
127.0.0.1:6379> 
复制代码

Sure, these deletions have been preserved, recover data that has no three records to prove that is not affected by the initiative to shut down the profile.

In addition savethere are other ways to save it?

save and save bgsave

Some, Redis offers saveand bgsavethese two different ways of saving, and the two methods will be called when the execution of rdbSavefunctions, but they are called in different ways:

save Call direct rdbSavemethod, blocking the main Redis process until save is complete. During the main process blocked, the server can not handle any client requests.
bgsave The fork a child process, the child process is responsible for calling rdbSave , and sends a signal to the main process after the save is complete, save the notification has been completed. Since rdbSave being called in the child process, so Redis server bgsave during execution can continue to process client requests.

save Is synchronous, bgsave asynchronous operation.

bgsaveUsage and commands saveusing the command method is the same:

127.0.0.1:6379> keys *
1) "lci"
2) "lcj"
3) "lcd"
4) "lcg"
5) "lcf"
6) "lce"
7) "lch"
127.0.0.1:6379> del lci lcj 
(integer) 2
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379> keys *
1) "lcd"
2) "lcg"
3) "lcf"
4) "lce"
5) "lch"
127.0.0.1:6379> 
复制代码

Save shutdown

In fact, the shutdowncommand also can save the data, the surprise is not a surprise. It will save the data before closing down, meaning not an accident?

127.0.0.1:6379> set app 1
OK
127.0.0.1:6379> set apps 1
OK
127.0.0.1:6379> keys *
1) "apps"
2) "lcd"
3) "lcg"
4) "lcf"
5) "app"
6) "lce"
7) "lch"
127.0.0.1:6379> shutdown
not connected> quit
gannicus@$ 
复制代码

Then Redis service was shut off. We need to restart Redis service to the client to see whether the effect:

gannicus@$ src/redis-cli
127.0.0.1:6379> keys *
1) "lce"
2) "lcf"
3) "lcd"
4) "lch"
5) "lcg"
复制代码

Not even take effect, thorn does not stimulate? Why is this? Obviously shutdown of official documents say will save the only exit, you lie ~

We note that there is a document

Suddenly realized that in the case of persistence to be opened by shutdowncommand closes it will not lose data, then to those few in the configuration file saveof configuration items to open it:

#   save ""
save 900 1
save 300 10
save 60 10000
复制代码

Then turn Redis service, and then try it again (process: Add -> shutdown -> Restart Services -> View):

127.0.0.1:6379> set app 1
OK
127.0.0.1:6379> set apps 1
OK
127.0.0.1:6379> shutdown
not connected> quit
gannicus@$ src/redis-cli
127.0.0.1:6379> keys *
1) "lce"
2) "lch"
3) "app" 4) "lcf" 5) "apps" 6) "lcd" 7) "lcg" 127.0.0.1:6379> 复制代码

He was finally figured out.

AOF way persistent and open configuration

Open AOF

AOF is not turned on by default, and if you want to enable you need to redis.conf configure the file open, open redis.conf:

$ vim redis.conf
复制代码

Then found in the file appendonly and no read yes:

appendonly yes
复制代码

It is the way to open the persistence of AOF.

Setting synchronous mode

AOF also supports several synchronous mode, they are:

appendfsync always  # 每次有数据修改发生时都会写入AOF文件（安全但是费时）。
appendfsync everysec  # 每秒钟同步一次，该策略为AOF的缺省策略。
appendfsync no  # 从不同步。高效但是数据不会被持久化。
复制代码

The default configuration everysec, you can be adjusted according to demand, and here I will change the configuration always:

appendfsync always
# appendfsync everysec
# appendfsync no
复制代码

Custom AOF record file name

Redis is provided with a default file name, as displayed in the configuration:

appendfilename "appendonly.aof"
复制代码

You can let it keep the default name, you can also specify a different file name, for example:

appendfilename "RNGLetme.aof"
复制代码

Will appendonly , appendfsync and appendfilename set up and save. Restart Redis service:

$./redis-server
复制代码

Command ls to view local files, you can see new life into a named RNGLetme.aof file, you can use:

$cat RNGLetme.aof  
复制代码

To view the contents inside, due to the change of the data is not currently carried out, it is blank.

Then open Redis clients:

$./redis-cli
复制代码

And adds several data records:

127.0.0.1:6379> set rng lpl
OK
127.0.0.1:6379> set ig lpl
OK
127.0.0.1:6379> set edg lpl
OK
127.0.0.1:6379> keys *
1) "edg"
2) "rng"
3) "ig"
127.0.0.1:6379> 
复制代码

It can be seen successfully added rng, edg, ig these three records, and then open the RNGLetme.aof file, look inside the record:

*2
$6
SELECT
$1
0
*3
$3
set
$3
rng
$3
lpl
*3
$3
set
$2
ig
$3
lpl
*3
$3
set
$3
edg
$3
lpl
复制代码

Each time data is added are recorded.

If that is the delete operation does, it will be recorded?

127.0.0.1:6379> del edg
(integer) 1
127.0.0.1:6379> keys *
1) "rng"
2) "ig"
127.0.0.1:6379> 
复制代码

After a delete operation, then look at the RNGLetme.aof records in the file:

*2
$6
SELECT
$1
0
*3
$3
set
$3
rng
$3
lpl
*3
$3
set
$2
ig
$3
lpl
*3
$3
set
$3
edg
$3
lpl
*2
$3
del
$3
edg
复制代码

Before recording contrast, the new del edg operating record. This confirms the previous description of the AOF: A log of changes in the way data is recorded.

AOF recovery testing

Here also by kill abnormal shutdown command to simulate Redis:

gannicus@$ kill -9 22645
复制代码

And then restart Redis service:

$ src/redis-server redis.conf
复制代码

Then take a look through the client, whether those data are:

$ src/redis-cli
127.0.0.1:6379> keys *
1) "ig"
2) "rng"
复制代码

Can be seen, rng and ig are still, it means that persistence is in force.

How to switch from RDB way to AOF way

In Redis version 2.2 or above, can be restarted without switching from the RDB to AOF:

Create a backup copy of the latest dump.rdb files will be backed up in a safe place. Execute the following two commands:

redis-cli config set appendonly yes
redis-cli config set save “”
复制代码

Ensure that the write command is correctly appended to the end AOF file. The first command opens up AOF features: Redis will block until the file is created until the completion of the initial AOF, after Redis will continue to process the request command and starts a write command appended to the end AOF file.

The second command execution function for closing RDB. This step is optional, if you prefer, you can also use the RDB and AOF both persistence functions simultaneously.

重要：别忘了在 redis.conf 中打开 AOF 功能！否则服务器重启后，之前通过 CONFIG SET 命令设置的配置就会被遗忘，程序会按原来的配置来启动服务器。

优先选择 RDB 还是 AOF 呢？

分析对比两种方式并做了测试后，发现这是两种不同风格的持久化方式，那么应该如何选择呢？

对于企业级的中大型应用，如果不想牺牲数据完整性但是又希望保持高效率，那么你应该同时使用 RDB 和 AOF 两种方式；
如果你不打算耗费精力在这个地方，只需要保证数据完整性，那么优先考虑使用 AOF 方式；
RDB 方式非常适合大规模的数据恢复，如果业务对数据完整性和一致性要求不高，RDB是很好的选择。

备份redis数据的建议

确保你的数据有完整的备份，磁盘故障、节点失效等问题问题可能让你的数据消失不见，不进行备份是非常危险的。

Redis 对于数据备份是非常友好的，因为你可以在服务器运行的时候对 RDB 文件进行复制： RDB 文件一旦被创建，就不会进行任何修改。当服务器要创建一个新的 RDB 文件时，它先将文件的内容保存在一个临时文件里面，当临时文件写入完毕时，程序才使用 rename(2) 原子地用临时文件替换原来的 RDB 文件。

这也就是说，无论何时，复制 RDB 文件都是绝对安全的。

创建一个定期任务（ cron job ），每小时将一个 RDB 文件备份到一个文件夹，并且每天将一个 RDB 文件备份到另一个文件夹。
确保快照的备份都带有相应的日期和时间信息，每次执行定期任务脚本时，使用 find 命令来删除过期的快照：比如说，你可以保留最近 48 小时内的每小时快照，还可以保留最近一两个月的每日快照。
至少每天一次，将 RDB 备份到你的数据中心之外，或者至少是备份到你运行 Redis 服务器的物理机器之外。

Redis 密码持久化

在 Redis 中数据需要持久化，密码也要持久化。在客户端通过命令：

config set requirepass zxc9527
复制代码

可以为 Redis 设置值为zxc9527的密码，但是当 Redis 关闭并重新启动后，权限验证功能就会失效，再也不需要密码。所以，密码也需要在 redis.conf 中持久化。打开 redis.conf 找到 requirepass 配置项，取消其注释并在后面设置密码：

requirepass zxc9527
复制代码

保存后重启 Redis 服务，密码持久化即生效。

转自：https://juejin.im/post/5bcab6f46fb9a05d3c802ea6

The persistence mechanism redis