Redis persistence

RDB snapshot (snapshot)

By default, Redis saves the in-memory database snapshot in a binary file named dump.rdb.
You can set Redis to automatically save a data set when the condition "the data set has at least M changes within N seconds" is met.
For example, the following settings will cause Redis to automatically save a data set when it meets the condition of "at least 1000 keys have been changed within 60 seconds":

# save 60 1000 //To close RDB, just comment out all save strategies.

You can also manually execute commands to generate RDB snapshots. Enter the redis client and execute the command save or bgsave to generate a dump.rdb file. Each command execution will snapshot all redis memory into a new rdb file and overwrite the original rdb snapshot file. .

bgsave's copy-on-write (COW) mechanism
Redis uses the copy-on-write technology (COW) provided by the operating system to generate snapshots while still processing write commands normally. Simply put, the bgsave sub-process is generated by fork of the main thread and can share all memory data of the main thread. After the bgsave sub-process runs, it starts to read the memory data of the main thread and writes them to the RDB file. At this time, if the main thread also performs read operations on these data, then the main thread and the bgsave sub-process will not affect each other. However, if the main thread wants to modify a piece of data, then the piece of data will be copied to generate a copy of the data. Then, the bgsave sub-process will write this copy data to the RDB file, and during this process, the main thread can still directly modify the original data.
Comparison between save and bgsave:

Order	save	bgsave
IO type	Synchronize	asynchronous
Whether to block other redis commands	yes	No (there will be a brief blocking when the child process is generated to execute the fork function)
the complexity	O(n)	O(n)
advantage	Does not consume additional memory	Does not block client commands
shortcoming	Block client commands	Need to fork the child process and consume memory

The background for configuring the automatic generation of rdb files uses the bgsave method.

AOF（append-only file）

The snapshot function is not very durable: if Redis fails for some reason, the server will lose the data that was recently written and has not yet been saved in the snapshot.
Starting from version 1.1, Redis has added a completely durable persistence method: AOF persistence, which records each modified instruction into the file appendonly.aof (write it to the os cache first, and fsync it to the disk every once in a while). For
example Execute the command " set tacy 666 ", the following data will be recorded in the aof file

*3
$3
set
$5
tacy
$3
666

This is a resp protocol format data,

The number after the asterisk represents how many parameters the command has.
The number after the $ sign represents how many characters this parameter has.

Note that if you execute a set command with an expiration time, what is recorded in the aof file is not the original command executed, but the timestamp of the key expiration. For example, if "
set lucy 888 ex 1000 " is executed, the corresponding record in the aof file is as follows

*3
$3
set
$6
lucy
$3
888
*3
$9
PEXPIREAT
$6
lucy
$13
1604249786301

You can turn on the AOF function by modifying the configuration file

# appendonly yes

From now on, whenever Redis executes a command that changes the data set (such as SET), the command will be appended to the end of the AOF file.
In this case, when Redis is restarted, the program can rebuild the data set by re-executing the commands in the AOF file.

You can configure how often Redis fsyncs data to disk.
There are three options:

appendfsync always：每次有新命令追加到 AOF 文件时就执行一次 fsync ，非常慢，也非常安全。
appendfsync everysec：每秒 fsync 一次，足够快，并且在故障时只会丢失 1 秒钟的数据。
appendfsync no：从不 fsync ，将数据交给操作系统来处理。更快，也更不安全的选择。

The recommended (and default) measure is to fsync once per second. This fsync strategy can balance speed and security.
AOF rewrite
There may be too many useless instructions in the AOF file, so AOF will regularly generate aof files based on the latest data in the memory.
For example, the following commands are executed:
Insert image description here

After rewriting, the AOF file becomes

*3
$3
SET
$2
readcount
$1
5

The following two configurations can control the AOF automatic rewrite frequency

# auto-aof-rewrite-min-size 64mb   //aof文件至少要达到64M才会自动重写，文件太小恢复速度本来就很快，重写的意义不大
# auto-aof-rewrite-percentage 100  //aof文件自上一次重写后文件大小增长了100%则再次触发重写

Of course, AOF can also be rewritten manually. Enter the redis client and execute the command bgrewriteaof to rewrite AOF.
Note that rewriting redis with AOF will fork a sub-process to do it (similar to the bgsave command), which will not have much impact on the normal command processing of redis.
Insert image description here

Both can be enabled in the production environment. If there are both rdb files and aof files when redis is started, the aof file will be preferred to restore data, because aof generally has more complete data.

Redis 4.0 hybrid persistence

When restarting Redis, we rarely use RDB to restore the memory state because a large amount of data will be lost. We usually use AOF log replay, but the performance of AOF log replay is much slower than RDB, so when the Redis instance is large, it takes a long time to start. To solve this problem, Redis 4.0 brings a new persistence option - hybrid persistence.
Hybrid persistence can be enabled through the following configuration (aof must be enabled first):

# aof-use-rdb-preamble yes

If hybrid persistence is turned on, when AOF is rewritten, it no longer simply converts the memory data into RESP commands and writes them to the AOF file. Instead, the memory before the moment of rewriting is processed as an RDB snapshot, and the RDB snapshot content and Incremental AOF commands to modify memory data exist together and are written into a new AOF file. The new file will not be called appendonly.aof at the beginning. It will not be renamed until the new AOF file is rewritten, overwriting the original AOF file. , complete the replacement of the old and new AOF files.
Therefore, when Redis restarts, the contents of the RDB can be loaded first, and then the incremental AOF log can be replayed to completely replace the previous AOF full file replay, so the restart efficiency is greatly improved.

The hybrid persistent AOF file structure is as follows
Insert image description here

Redis data backup strategy:

Write a crontab scheduled scheduling script to copy a backup of RDB or AOF to a directory every hour, and only keep the backup of the last 48 hours.
Keep a backup of the current day's data in a directory every day. You can keep the backup of the last month.
Every time I copy a backup, I delete the backup that is too old.
Copy a backup of the current machine to other machines every night to prevent machine damage

Redis master-slave architecture

Insert image description here

Build redis master-slave architecture and configure slave nodes:

1. Copy a copy of the redis.conf file

2. Modify the relevant configuration to the following values:

port 6380
pidfile /var/run/redis_6380.pid  # 把pid进程号写入pidfile配置的文件
logfile "6380.log"
dir /usr/local/redis-5.0.3/data/6380  # 指定数据存放目录
# 需要注释掉bind
# bind 127.0.0.1（bind绑定的是自己机器网卡的ip，如果有多块网卡可以配多个ip，代表允许客户端通过机器的哪些网卡ip去访问，内网一般可以不配置bind，注释掉即可）

3. Configure master-slave replication

replicaof 192.168.0.60 6379   # 从本机6379的redis实例复制数据，Redis 5.0之前使用slaveof
replica-read-only yes  # 配置从节点只读

4. Start the slave node

redis-server redis.conf   # redis.conf文件务必用你复制并修改了之后的redis.conf文件

5. Connect the slave node

redis-cli -p 6380

6. Test writing data on the 6379 instance and see whether the 6380 instance can synchronize the newly modified data in time.

7. You can configure a 6381 slave node yourself

Redis master-slave working principle

If you configure a slave for the master, regardless of whether the slave connects to the master for the first time, it will send a PSYNC command to the master to request copy data.
After receiving the PSYNC command, the master will perform data persistence in the background and generate the latest RDB snapshot file through bgsave. During the persistence period, the master will continue to receive client requests, and it will cache these requests that may modify the data set in memory. When the persistence is completed, the master will send the RDB file data set to the slave, and the slave will persist the received data to generate RDB, and then load it into the memory. Then, the master sends the previously cached commands in memory to the slave.
When the connection between the master and the slave is disconnected for some reason, the slave can automatically reconnect to the master. If the master receives multiple slave concurrent connection requests, it will only persist once, not once for each connection, and then Then send this persistent data to multiple concurrently connected slaves.