NoSQL-Redis persistence

1. Redis High Availability:

1 Overview:

(1) In the web server, high availability refers to the time when the server can be accessed normally, and the measure is how long the normal service can be provided (99.9%, 99.99%, 99.999%, etc.).

(2) In the context of Redis, the meaning of high availability seems to be broader. In addition to ensuring the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity and data security without loss.

(3) In Redis, the technologies to achieve high availability mainly include persistence, master-slave replication, sentry and cluster clusters. The following describes their functions and what problems they solve.

① Persistence: Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). Its main function is data backup, that is, storing data on the hard disk to ensure that data will not be lost due to process exit.

② Master-slave replication: Master-slave replication is the basis of highly available Redis. Sentinels and clusters are based on master-slave replication to achieve high availability. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing for read operations and simple fault recovery. Defects: Failure recovery cannot be automated; write operations cannot be load-balanced; storage capacity is limited by a single machine.

③ Sentinel: Based on master-slave replication, Sentinel realizes automatic fault recovery. Defects: Write operations cannot be load-balanced; storage capacity is limited by a single machine.

④ Cluster cluster: Through the cluster, Redis solves the problems that the write operation cannot be load-balanced and the storage capacity is limited by a single machine, and realizes a relatively complete high-availability solution.

Second, Redis persistence:

1. Persistence function:

Redis is an in-memory database, and the data is stored in the memory. In order to avoid the permanent loss of data after the Redis process exits abnormally due to server power failure and other reasons, it is necessary to periodically save the data in Redis from the memory in some form (data or commands) to the hard disk; when Redis restarts next time, use the persistent file to achieve data recovery. Additionally, the persistent files can be copied to a remote location for disaster backup purposes.

2. Redis provides two methods for persistence:

(1) RDB persistence: The principle is to save the database records of Reids in memory to disk regularly.
(2) AOF persistence (append only file): The principle is to write the operation log of Reids to the file in an appended way, similar to MySQL's binlog.

Because the real-time performance of AOF persistence is better, that is, less data is lost when the process exits unexpectedly, so AOF is currently the mainstream persistence method, but RDB persistence still has its place.

3. RDB persistence:

1. Definition:

RDB persistence refers to saving the snapshot of the data in the current process in the memory to the hard disk within a specified time interval (so it is also called snapshot persistence), stored in binary compression, and the saved file suffix is rdb; when Redis restarts , you can read the snapshot file to restore data.

2. Trigger conditions:

The triggering of RDB persistence is divided into manual triggering and automatic triggering.

(1) Manual trigger:
Both the save command and the bgsave command can generate RDB files.
The save command will block the Redis server process until the RDB file is created. During the blockage of the Redis server, the server cannot process any command requests.
The bgsave command creates a child process, which is responsible for creating the RDB file, and the parent process (that is, the Redis main process) continues to process requests.

During the execution of the bgsave command, only the fork child process will block the server, but for the save command, the entire process will block the server, so save has been basically abandoned, and the use of save must be avoided in the online environment.

(2) Automatic trigger:
When automatically triggering RDB persistence, Redis will also choose bgsave instead of save for persistence.

The most common case of automatic triggering of save mn
is to pass save mn in the configuration file to specify that bgsave will be triggered when n changes occur within m seconds.

vim /etc/redis/6379.conf
--219行--以下三个save条件满足任意一个时，都会引起bgsave的调用
save 900 1 ：当时间到900秒时，如果redis数据发生了至少1次变化，则执行bgsave
save 300 10 ：当时间到300秒时，如果redis数据发生了至少10次变化，则执行bgsave
save 60 10000 ：当时间到60秒时，如果redis数据发生了至少10000次变化，则执行bgsave
--254行--指定RDB文件名
dbfilename dump.rdb
--264行--指定RDB文件和AOF文件所在目录
dir /var/lib/redis/6379
--242行--是否开启RDB文件压缩
rdbcompression yes

(3) Other automatic trigger mechanisms:
In addition to save mn, there are other situations that trigger bgsave:
① In the master-slave replication scenario, if the slave node performs a full copy operation, the master node will execute the bgsave command and save the rdb file sent to the slave node.
② When the shutdown command is executed, rdb persistence is automatically executed.

3. Execution process:

(1) The Redis parent process first judges: whether it is currently executing save, or the child process of bgsave/bgrewriteaof, if it is executing, the bgsave command will return directly. The child processes of bgsave/bgrewriteaof cannot be executed at the same time, mainly based on performance considerations: two concurrent child processes perform a large number of disk write operations at the same time, which may cause serious performance problems.
(2) The parent process executes the fork operation to create a child process. During this process, the parent process is blocked, and Redis cannot execute any commands from the client. (3) After the parent process forks, the bgsave command returns the "
Background saving started" message and no longer blocks The parent process can respond to other commands
(4) The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion
(5) The child process sends a signal to the parent process to indicate completion, and the parent process Process Update Statistics

insert image description here

4. Load at startup:

(1) The loading of the RDB file is automatically executed when the server starts, and there is no special command. But because AOF has a higher priority, when AOF is turned on, Redis will be loaded first.

(2) AOF files are used to restore data; only when AOF is closed, RDB files will be detected and automatically loaded when the Redis server starts. The server is blocked while loading the RDB file until the loading is complete.

(3) When Redis loads the RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log, and Redis will fail to start.

5. Advantages and disadvantages of RDB:

(1) Disadvantages:
① Data integrity is not as good as aof
② rdb is similar to snapshot (perfect)
③ When performing backup, it will block the process
(2) Advantages:
① Persistent speed block (because of the saved data result), when writing to * ．RDB persistent files are compressed to reduce their size
② In the cluster, redis master-slave replication, from-master server for synchronization, by default, RDB files are used first, recovery operations, all synchronization performance is high

4. AOF persistence:

RDB persistence is to write process data into files, while AOF persistence is to record each write and delete command executed by Redis into a separate log file, and the query operation will not be recorded; when Redis restarts, execute the AOF file again command in to restore data. Compared with RDB, AOF has better real-time performance, so it has become the mainstream persistence solution.

1. Turn on AOF:

The Redis server enables RDB by default and disables AOF; to enable AOF, it needs to be configured in the configuration file:

vim /etc/redis/6379.conf
--700行--修改，开启AOF
appendonly yes
--704行--指定AOF文件名称
appendfilename "appendonly.aof"
--796行--是否忽略最后一条可能存在问题的指令
aof-load-truncated yes
/etc/init.d/redis_6379 restart

2. Execution process:

Since each write command of Redis needs to be recorded, AOF does not need to be triggered. The following describes the execution process of AOF.

(1) command append (append):
① Redis first appends the write command to the buffer instead of directly writing the file, mainly to avoid writing directly to the hard disk every time there is a write command, causing the hard disk IO to become the bottleneck of Redis load .
② The format of command addition is the protocol format of Redis command request. It is a plain text format, which has the advantages of good compatibility, strong readability, easy processing, simple operation and avoiding secondary overhead.

③ In the AOF file, except for the select command used to specify the database (such as select 0 to select database 0), which is added by Redis, all others are write commands sent by the client.

(2) File writing (write) and file synchronization (sync):
Redis provides a variety of synchronization file strategies for the AOF buffer area. The strategy involves the write function and fsync function of the operating system. The description is as follows:
In order to improve file writing efficiency , in a modern operating system, when the user calls the write function to write data into a file, the operating system usually temporarily stores the data in a memory buffer, and only when the buffer is full or exceeds the specified time limit, the operating system will actually store the data The data in the buffer is written to the hard disk. Although this kind of operation improves efficiency, it also brings security problems: if the computer shuts down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately update the data in the buffer. The data is written to the hard disk to ensure data security.

There are three synchronization methods for the synchronization file strategy of the AOF cache area, which are:

vim /etc/redis/6379.conf
--729--
●appendfsync always： 命令写入aof_buf后立即调用系统fsync操作同步到AOF文件，fsync完成后线程返回。这种情况下，每次有写命令都要同步到AOF文件，硬盘IO成为性能瓶颈，Redis只能支持大约几百TPS写入，严重降低了Redis的性能；即便是使用固态硬盘（SSD），每秒大约也只能处理几万个命令，而且会大大降低SSD的寿命。

●appendfsync no： 命令写入aof_buf后调用系统write操作，不对AOF文件做fsync同步；同步由操作系统负责，通常同步周期为30秒。这种情况下，文件同步的时间不可控，且缓冲区中堆积的数据会很多，数据安全性无法保证。

●appendfsync everysec： 命令写入aof_buf后调用系统write操作，write完成后线程返回；fsync同步文件操作由专门的线程每秒调用一次。everysec是前述两种策略的折中，是性能和数据安全性的平衡，因此是Redis的默认配置，也是我们推荐的配置。

(3) File rewriting (rewrite)

① File rewriting refers to rewriting AOF files regularly to reduce the size of AOF files. It should be noted that AOF rewriting is to convert the data in the Redis process into write commands, synchronize to the new AOF file, and will not perform any read or write operations on the old AOF file!

② Another point to note about file rewriting is: for AOF persistence, although file rewriting is strongly recommended, it is not necessary; even without file rewriting, data can be persisted and started in Redis Import at time; therefore, in some realities, automatic file rewriting will be turned off, and then scheduled to be executed at a certain time every day through a scheduled task.

③ The reason why file rewriting can compress AOF files is:

Expired data is no longer written to the file
Invalid commands are no longer written to the file: for example, some data is repeatedly set (set mykey v1, set mykey v2), some data is deleted (set myset v1, del myset), etc.
Multiple commands can be combined into one: for example, sadd myset v1, sadd myset v2, sadd myset v3 can be combined into sadd myset v1 v2 v3.
From the above content, it can be seen that since the commands executed by AOF are reduced after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery speed.

④ The trigger of file rewriting is divided into manual trigger and automatic trigger:

Manual trigger: directly call the bgrewriteaof command, the execution of this command is somewhat similar to bgsave: both fork sub-processes perform specific work, and both are blocked only when fork.
Automatic trigger: BGREWRITEAOF is automatically executed by setting the auto-aof-rewrite-min-size option and the auto-aof-rewrite-percentage option. Only when the two options of auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are satisfied at the same time, AOF rewriting will be automatically triggered, that is, the bgrewriteaof operation.

vim /etc/redis/6379.conf
--729--
●auto-aof-rewrite-percentage 100	：当前AOF文件大小(即aof_current_size)是上次日志重写时AOF文件大小(aof_base_size)两倍时，发生BGREWRITEAOF操作
●auto-aof-rewrite-min-size 64mb ：当前AOF文件执行BGREWRITEAOF命令的最小值，避免刚开始启动Reids时由于文件尺寸较小导致频繁的BGREWRITEAOF	


关于文件重写的流程，有两点需要特别注意：(1)重写由父进程fork子进程进行；(2)重写期间Redis执行的写命令，需要追加到新的AOF文件中，为此Redis引入了aof_rewrite_buf缓存。

⑤ The process of file rewriting is as follows:
(1) The Redis parent process first judges whether there is a child process currently executing bgsave/bgrewriteaof. If it exists, the bgrewriteaof command returns directly. If there is a bgsave command, wait for the bgsave execution to complete before executing it.
(2) The parent process executes the fork operation to create a child process, and the parent process is blocked during this process.
(3.1) After the parent process forks, the bgrewriteaof command returns the message "Background append only file rewrite started" and no longer blocks the parent process, and can respond to other commands. All write commands of Redis are still written into the AOF buffer, and are synchronized to the hard disk according to the appendfsync strategy to ensure the correctness of the original AOF mechanism.
(3.2) Since the fork operation uses the copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process is still responding to the command, Redis uses the AOF rewrite buffer (aof_rewrite_buf) to save this part of the data to prevent the loss of this part of the data during the generation of the new AOF file. That is to say, during the execution of bgrewriteaof, Redis write commands are simultaneously appended to two buffers, aof_buf and aof_rewrite_buf.
(4) According to the memory snapshot, the child process writes to the new AOF file according to the command merging rules.
(5.1) After the child process writes the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.
(5.2) The parent process writes the data in the AOF rewrite buffer to the new AOF file, thus ensuring that the database state saved in the new AOF file is consistent with the current state of the server.
(5.3) Replace the old file with the new AOF file to complete AOF rewriting.

insert image description here

3. Load at startup:

(1) When AOF is turned on, Redis will first load the AOF file to restore data when it starts; only when AOF is turned off, it will load the RDB file to restore data.

(2) When AOF is enabled but the AOF file does not exist, it will not be loaded even if the RDB file exists.

(3) When Redis loads the AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log, and Redis will fail to start. However, if the end of the AOF file is incomplete (sudden machine downtime, etc. may easily cause the end of the file to be incomplete), and the aof-load-truncated parameter is enabled, a warning will be output in the log, Redis ignores the end of the AOF file, and the startup is successful. The aof-load-truncated parameter is enabled by default.

4. Advantages and disadvantages of AOF:

(1) Disadvantages
① When the statement is executed all the time, the content of the AOF backup is larger, the content of the RDB backup is smaller, the result of the backup, the statement

② AOF consumes more performance and occupies more and more disks (equivalent to mysql backup)

(2) Advantages:

① The data integrity of AOF is higher than that of RDB

② The rewrite function will delete invalid statements (the purpose is to save the disk space occupied by AOF files)

5. Redis performance management:

1. View Redis memory usage:

info memory

2. Memory fragmentation rate:

(1) Calculated by dividing the used_memory_rss memory value allocated by the operating system by the total memory value used_memory used by Redis.
The memory value used_memory_rss indicates the size of the physical memory occupied by the process, that is, the size of the memory allocated to the Redis instance by the operating system.

(2) In addition to user-defined data and internal overhead, the used_memory_rss indicator also includes the overhead of memory fragmentation, which is caused by the inefficient allocation/reclaim of physical memory by the operating system (discontinuous physical memory allocation).

(3) For example: Redis needs to allocate continuous memory blocks to store 1G data sets. If there is no continuous memory block exceeding 1G in the physical memory, the operating system has to use multiple discontinuous small memory blocks to allocate and store the 1G data, and this operation will lead to memory fragmentation.

It is reasonable for the memory fragmentation rate to be slightly greater than 1. This value indicates that the memory fragmentation rate is relatively low, and it also shows that Redis does not exchange memory.
If the memory fragmentation rate exceeds 1.5, it means that Redis consumes 150% of the actual physical memory required, of which 50% is the memory fragmentation rate. You need to enter the shutdown save command on the redis-cli tool to let the Redis database execute the save operation and close the Redis service, and then restart the server.
If the memory fragmentation rate is lower than 1, it means that the Redis memory allocation exceeds the physical memory, and the operating system is exchanging memory. Need to increase available physical memory or reduce Redis memory footprint.

3. Memory usage:

If the memory usage of the redis instance exceeds the maximum available memory, the operating system will start exchanging memory and swap space.
Ways to avoid memory swapping:
- Choose to install the Redis instance for the size of the cached data
- Use Hash data structure storage as much as possible
- Set the expiration time of the key

4. Internal recovery key:

The memory cleaning strategy ensures reasonable allocation of redis limited memory resources.

When the set maximum threshold is reached, a key recycling strategy needs to be selected. By default, the recycling strategy is to prohibit deletion.

vim /etc/redis/6379.conf
--598--
maxmemory-policy noenviction
volatile-lru：使用LRU算法从已设置过期时间的数据集合中淘汰数据(移除最近最少使用的key，针对设置了TTL的key)
volatile-ttl：从已设置过期时间的数据集合中挑选即将过期的数据淘汰（移除最近过期的key）
volatile-random：从已设置过期时间的数据集合中随机挑选数据淘汰（在设置了TTL的key里随机移除）
allkeys-lru：使用LRU算法从所有数据集合中淘汰数据（移除最少使用的key，针对所有的key）
allkeys-random：从数据集合中任意选择数据淘汰（随机移除key）
noenviction：禁止淘汰数据（不删除直到写满时报错）

6. Summary of RDB and AOF:

1. Basic understanding of RDB and AOF:

(1) RDB: Periodically save the data in memory to disk.

(2) AOF: Synchronize the execution process to the disk from the execution operation log of redis.

2. The process of RDB and AOF persistence:

(1) RDB: ① In the memory, the method of writing to the disk to save.

② The result data is written to the data object saved in the disk.

③ After the memory is written to the disk, it will be compressed to reduce the disk space occupied by *.rdb.

(2) AOF: ① In the memory, append is added to the buffer, and the CPU resource is called to write to the disk.

② The execution statement in the operation log is appended to the buffer, and the CPU resource is called to write to the disk.

③ Memory, buffer, and disk will be rewritten periodically after writing, skipping some "invalid operations" to save.

3. The triggering method of RDB and AOF:

(1) RDB: ① Manual trigger

② Automatic trigger: save mn.

③ Special trigger: When manually closed, rdb will be persisted. /etc/init.d/redis_6379 restart

(2) AOF: ① Manual trigger

② Automatic trigger: always execute each statement synchronously for persistence (scenario with strong consistency)

no never persist

every second AOE persistence is performed every second (recommended, balanced scenario)

4. RDB and AOE priority:

(1) Because redis stores data in memory by default, the data in memory will be lost if redis is started or closed

(2) Every time redis starts, it will read the persistent file and send the data to the memory to ensure the integrity of the reids data

(3) RDB and AOF priority aof> RDB configuration file can specify to start modifying each parameter 6379.conf

5. Advantages and disadvantages of RDB and AOF:

(1) RDB persistence:

① Advantages: RDB files are compact, small in size, fast in network transmission, suitable for full copying; recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB compared to AOF is the relatively small impact on performance.

② Disadvantages: The fatal disadvantage of RDB files is that the persistence method of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable, so AOF persistence become mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, old versions of Redis are not compatible with new versions of RDB files).

③ For RDB persistence, on the one hand, the Redis main process will be blocked when bgsave performs the fork operation, and on the other hand, writing data to the hard disk by the child process will also bring IO pressure.

(2) AOF persistence

① Corresponding to RDB persistence, the advantage of AOF is that it supports second-level persistence and good compatibility. The disadvantage is that the file is large, the recovery speed is slow, and it has a great impact on performance.

② For AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under the everysec strategy), the IO pressure is greater, and it may even cause AOF additional blocking problems.

The method determines that real-time persistence cannot be achieved, and today when data is becoming more and more important, a large amount of data loss is often unacceptable, so AOF persistence has become the mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, old versions of Redis are not compatible with new versions of RDB files).

(2) AOF persistence

③ The rewriting of the AOF file is similar to the bgsave of the RDB, and there will be blocking during fork and IO pressure of the child process. Relatively speaking, since AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.