In-depth study of Redis: persistence

1. Overview of Redis high availability

Before introducing Redis high availability, first explain the meaning of high availability in the context of Redis.

We know that in web servers, high availability refers to the time when the server can be accessed normally, and the measure is how long normal services can be provided (99.9%, 99.99%, 99.999%, etc.). However, in the context of Redis, the meaning of high availability seems to be broader. In addition to ensuring the provision of normal services (such as master-slave separation, rapid disaster recovery technology), it is also necessary to consider the expansion of data capacity and data security without loss.

In Redis, the technologies to achieve high availability mainly include persistence, replication, sentinels and clusters. The following describes their functions and what problems they solve.

1. Persistence: Persistence is the simplest high-availability method (sometimes not even classified as a high-availability method). Its main function is data backup, that is, storing data on the hard disk to ensure that data will not be lost due to process exit.
2. Replication: Replication is the basis of highly available Redis. Sentinels and clusters are based on replication to achieve high availability. Replication mainly implements multi-machine backup of data, as well as load balancing for read operations and simple fault recovery. Defects: Failure recovery cannot be automated; write operations cannot be load-balanced; storage capacity is limited by a single machine.
3. Sentinel: Based on replication, Sentinel implements automated failure recovery. Defects: Write operations cannot be load-balanced; storage capacity is limited by a single machine.
4. Cluster: Through clustering, Redis solves the problem that write operations cannot be load-balanced, and the storage capacity is limited by a single machine, and realizes a relatively complete high-availability solution.

2. Overview of Redis Persistence

Persistence function: Redis is an in-memory database, and data is stored in memory. In order to avoid permanent loss of data caused by process exit, it is necessary to periodically save the data in Redis in some form (data or command) from memory to hard disk; When Redis restarts next time, use the persistent file to achieve data recovery. Additionally, the persistent files can be copied to a remote location for disaster backup purposes.

Redis persistence is divided into RDB persistence and AOF persistence : the former saves the current data to the hard disk, and the latter saves each executed write command to the hard disk (similar to MySQL's binlog); due to the real-time nature of AOF persistence Better, that is, less data is lost when the process exits unexpectedly, so AOF is currently the mainstream persistence method, but RDB persistence still has its place.

The following introduces RDB persistence and AOF persistence in turn; due to the differences between Redis versions, Redis 3.0 shall prevail unless otherwise specified.

3. RDB Persistence

RDB persistence is to generate a snapshot of the data in the current process and save it to the hard disk (so it is also called snapshot persistence). The suffix of the saved file is rdb; when Redis restarts, the snapshot file can be read to restore the data.

1. Trigger conditions

The triggering of RDB persistence is divided into manual triggering and automatic triggering.

1) Manual trigger

Both the save command and the bgsave command can generate RDB files.

The save command will block the Redis server process until the RDB file is created. During the blockage of the Redis server, the server cannot process any command requests.

The bgsave command creates a child process, which is responsible for creating the RDB file, and the parent process (that is, the Redis main process) continues to process requests.

At this time, the server execution log is as follows:

During the execution of the bgsave command, only the fork child process will block the server, but for the save command, the entire process will block the server, so save has been basically abandoned, and the use of save should be avoided in the online environment; only the bgsave command will be introduced later . In addition, when automatically triggering RDB persistence, Redis will also choose bgsave instead of save for persistence; the conditions for automatically triggering RDB persistence are introduced below.

2) Automatic trigger

save m n

The most common case of automatic triggering is to pass save mn in the configuration file to specify that bgsave will be triggered when n changes occur within m seconds.

For example, if you view the default configuration file of redis (redis.conf in the root directory of redis under Linux), you can see the following configuration information:

The meaning of save 900 1 is: when the time reaches 900 seconds, if the redis data has changed at least once, execute bgsave; save 300 10 and save 60 10000 are the same. When any one of the three save conditions is met, bgsave will be called.

The realization principle of save mn

Redis's save mn is implemented through the serverCron function, dirty counter, and lastsave timestamp.

serverCron is a periodic operation function of the Redis server, which is executed every 100ms by default; this function maintains the state of the server, and one of its tasks is to check whether the conditions of the save mn configuration are met, and if so, execute bgsave.

The dirty counter is a state maintained by the Redis server, which records how many times the server state has been modified (including additions, deletions, and modifications) since the last execution of the bgsave/save command; and when save/bgsave is executed, dirty will be reset to 0 .

For example, if Redis executes set mykey helloworld, the dirty value will be +1; if sadd myset v1 v2 v3 is executed, the dirty value will be +3; note that dirty records how many times the server has modified, not the client execution How many commands to modify the data have been executed.

The lastsave timestamp is also a state maintained by the Redis server, which records the time when save/bgsave was successfully executed last time.

The principle of save mn is as follows: every 100ms, the serverCron function is executed; in the serverCron function, the save conditions configured by save mn are traversed, and bgsave is performed as long as one condition is met. For each save mn condition, it is only satisfied when the following two conditions are met at the same time:

(1) Current time - lastsave > m

（2）dirty >= n

save mn execution log

The following figure shows the server printing logs when save mn triggers bgsave execution:

Other automatic trigger mechanisms

In addition to save mn, there are some other situations that trigger bgsave:

• In the master-slave replication scenario, if the slave node performs a full copy operation, the master node will execute the bgsave command and send the rdb file to the slave node
• When the shutdown command is executed, RDB persistence is automatically executed, as shown in the following figure:

2. Execution process

The conditions for triggering bgsave are introduced above, and the execution process of the bgsave command will be explained below, as shown in the figure below (picture source: https://blog.csdn.net/a1007720052/article/details/79126253):

The operations performed in the 5 steps in the picture are as follows:

1) The Redis parent process first judges: whether it is currently executing save, or a child process of bgsave/bgrewriteaof (the command will be described in detail later), if it is executing, the bgsave command returns directly. The child processes of bgsave/bgrewriteaof cannot be executed at the same time, mainly based on performance considerations: two concurrent child processes perform a large number of disk write operations at the same time, which may cause serious performance problems.

2) The parent process executes the fork operation to create a child process. During this process, the parent process is blocked, and Redis cannot execute any commands from the client

3) After the parent process forks, the bgsave command returns the message "Background saving started" and no longer blocks the parent process, and can respond to other commands

4) The child process creates an RDB file, generates a temporary snapshot file based on the memory snapshot of the parent process, and atomically replaces the original file after completion

5) The child process sends a signal to the parent process to indicate completion, and the parent process updates the statistics

3. RDB file

RDB files are compressed binary files, and some details about RDB files are introduced below.

Storage path

The storage path of RDB files can be configured before startup, or dynamically set by commands.

Configuration: dir configures the specified directory, and dbfilename specifies the file name. The default is the dump.rdb file in the Redis root directory.

Dynamic setting: After Redis starts, the RDB storage path can also be dynamically modified, which is very useful when the disk is damaged or the space is insufficient; the execution commands are config set dir {newdir} and config set dbfilename {newFileName}. As shown below (Windows environment):

RDB file format

The RDB file format is shown in the figure below (picture source: "Redis Design and Implementation"):

The meaning of each field is as follows:

1) REDIS: constant, saving 5 characters of "REDIS".

2) db_version: The version number of the RDB file, note that it is not the version number of Redis.

3) SELECTDB 0 pairs: indicates a complete database (No. 0 database), similarly SELECTDB 3 pairs indicates a complete No. 3 database; only when there are key-value pairs in the database, there will be information about the database in the RDB file ( In Redis shown in the figure above, only databases 0 and 3 have key-value pairs); if all databases in Redis do not have key-value pairs, this part is omitted directly. Among them: SELECTDB is a constant, which means that it is followed by the database number; 0 and 3 are database numbers; pairs store specific key-value pair information, including key, value, and its data type, internal encoding, expiration time, Compress information and more.

4) EOF: constant, marking the end of the RDB file text content.

5) check_sum: the checksum of all the previous content; when Redis loads the RBD file, it will calculate the previous checksum and compare it with the check_sum value to determine whether the file is damaged.

compression

Redis uses the LZF algorithm to compress RDB files by default. Although compression is time-consuming, it can greatly reduce the size of RDB files, so compression is enabled by default; it can be disabled by command:

It should be noted that the compression of the RDB file is not performed on the entire file, but on the strings in the database, and only when the string reaches a certain length (20 bytes).

4. Load at startup

The loading of the RDB file is automatically executed when the server starts, and there is no special command. However, because AOF has a higher priority, when AOF is enabled, Redis will prioritize loading AOF files to restore data; only when AOF is disabled, RDB files will be detected and automatically loaded when the Redis server starts. The server is blocked while loading the RDB file until the loading is complete.

The execution of automatic loading can be seen in the Redis startup log:

When Redis loads the RDB file, it will verify the RDB file. If the file is damaged, an error will be printed in the log, and Redis will fail to start.

5. Summary of common RDB configurations

The following are commonly used configuration items for RDB and their default values; those introduced earlier will not be described in detail here.

• save mn: The condition for automatic triggering of bgsave; if there is no save mn configuration, it is equivalent to automatic RDB persistence shutdown, but it can still be triggered by other methods at this time
• stop-writes-on-bgsave-error yes: When an error occurs in bgsave, whether Redis stops executing the write command; if set to yes, when there is a problem with the hard disk, it can be found in time to avoid a large amount of data loss; if it is set to no, then Redis ignores the error of bgsave and continues to execute the write command. When monitoring the system (especially the hard disk) of the Redis server, this option is considered to be set to no
• rdbcompression yes: Whether to enable RDB file compression
• rdbchecksum yes: Whether to enable the verification of RDB files, it works when writing files and reading files; turning off checksum can bring about 10% performance improvement when writing files and starting files, but it cannot be found when data is damaged
• dbfilename dump.rdb: RDB file name
• dir ./: the directory where RDB files and AOF files are located

Four, AOF persistence

RDB persistence is to write process data into files, and AOF persistence (that is, Append Only File persistence) is to record each write command executed by Redis into a separate log file (a bit like MySQL's binlog); when When Redis restarts, execute the commands in the AOF file again to restore the data.

Compared with RDB, AOF has better real-time performance, so it has become the mainstream persistence solution.

1. Enable AOF

The Redis server enables RDB by default and disables AOF; to enable AOF, it needs to be configured in the configuration file:

appendonly yes

2. Execution process

Since each write command of Redis needs to be recorded, AOF does not need to be triggered. The following describes the execution process of AOF.

The execution process of AOF includes:

• Command append (append): Append the Redis write command to the buffer aof_buf;
• File writing (write) and file synchronization (sync): Synchronize the content in aof_buf to the hard disk according to different synchronization strategies;
• File rewrite (rewrite): Periodically rewrite the AOF file to achieve the purpose of compression.

1) Instruction addition (append)

Redis first appends the write command to the buffer instead of directly writing the file, mainly to avoid writing the command directly to the hard disk every time, causing the hard disk IO to become the bottleneck of Redis load.

The format of command append is the protocol format of Redis command request, which is a plain text format, which has the advantages of good compatibility, strong readability, easy processing, simple operation and avoiding secondary overhead; the specific format is omitted. In the AOF file, except for the select command used to specify the database (such as select 0 to select database 0), which is added by Redis, all others are write commands sent by the client.

2) File writing (write) and file synchronization (sync)

Redis provides a variety of synchronization file strategies for the AOF cache area. The strategy involves the write function and fsync function of the operating system, as follows:

In order to improve file writing efficiency, in modern operating systems, when a user calls the write function to write data into a file, the operating system usually temporarily stores the data in a memory buffer. When the buffer is full or exceeds the specified After the time limit, the data in the buffer is actually written to the hard disk. Although this kind of operation improves efficiency, it also brings security problems: if the computer shuts down, the data in the memory buffer will be lost; therefore, the system also provides synchronization functions such as fsync and fdatasync, which can force the operating system to immediately update the data in the buffer. The data is written to the hard disk to ensure data security.

The synchronization file policy of the AOF cache is controlled by the parameter appendfsync, and the meanings of each value are as follows:

• always: call the system fsync operation to synchronize to the AOF file immediately after the command is written into aof_buf, and the thread returns after fsync is completed. In this case, every write command must be synchronized to the AOF file, and hard disk IO becomes a performance bottleneck. Redis can only support about a few hundred TPS writes, which seriously reduces the performance of Redis; even if a solid state drive (SSD) is used, It can only process tens of thousands of commands per second, and it will greatly reduce the life of SSD.
• no: Call the system write operation after the command is written into aof_buf, and do not perform fsync synchronization on the AOF file; the synchronization is handled by the operating system, and the synchronization period is usually 30 seconds. In this case, the timing of file synchronization is uncontrollable, and there will be a lot of data accumulated in the buffer, so data security cannot be guaranteed.
• everysec: After the command is written into aof_buf, the system write operation is called, and the thread returns after the write is completed; the fsync synchronization file operation is called once per second by a dedicated thread. Everysec is a compromise between the above two strategies, and it is a balance between performance and data security. Therefore, it is the default configuration of Redis and our recommended configuration.

3) File rewriting (rewrite)

As time goes by, the Redis server executes more and more write commands, and the AOF file will become larger and larger; too large AOF files will not only affect the normal operation of the server, but also cause data recovery to take too long.

File rewriting refers to periodically rewriting the AOF file to reduce the size of the AOF file. It should be noted that AOF rewriting is to convert the data in the Redis process into write commands and synchronize to the new AOF file; it will not perform any read or write operations on the old AOF file!

Another point to note about file rewriting: For AOF persistence, file rewriting is strongly recommended, but not necessary; even without file rewriting, data can be persisted and started in Redis Time to import; therefore, in some implementations, automatic file rewriting will be turned off, and then scheduled to be executed at a certain time every day through a scheduled task.

The reason why file rewriting can compress AOF files is:

• Expired data is no longer written to the file
• Invalid commands are no longer written to the file: for example, some data is repeatedly set (set mykey v1, set mykey v2), some data is deleted (sadd myset v1, del myset), etc.
• Multiple commands can be combined into one: for example, sadd myset v1, sadd myset v2, sadd myset v3 can be combined into sadd myset v1 v2 v3. However, in order to prevent client buffer overflow caused by a single command being too large, it is not necessary to use only one command for key types of list, set, hash, and zset; instead, the command is divided into multiple commands based on a certain constant. This constant is defined in redis.h/REDIS_AOF_REWRITE_ITEMS_PER_CMD and cannot be changed. The value in version 3.0 is 64.

From the above content, it can be seen that since the commands executed by AOF are reduced after rewriting, file rewriting can not only reduce the space occupied by the file, but also speed up the recovery speed.

File rewrite trigger

The trigger of file rewriting is divided into manual trigger and automatic trigger:

Manual trigger: directly call the bgrewriteaof command, the execution of this command is somewhat similar to bgsave: both fork sub-processes perform specific work, and both are blocked only when fork.

At this time, the server execution log is as follows:

Automatic trigger: Determine the trigger timing according to the auto-aof-rewrite-min-size and auto-aof-rewrite-percentage parameters, as well as the status of aof_current_size and aof_base_size.

• auto-aof-rewrite-min-size: When performing AOF rewrite, the minimum size of the file, the default value is 64MB.
• auto-aof-rewrite-percentage: When performing AOF rewriting, the ratio of the current AOF size (ie aof_current_size) to the AOF size (aof_base_size) at the time of the last rewrite.

Among them, the parameters can be viewed through the config get command:

The status can be viewed through info persistence:

Only when the two parameters of auto-aof-rewrite-min-size and auto-aof-rewrite-percentage are satisfied at the same time, AOF rewriting will be automatically triggered, that is, bgrewriteaof operation.

When bgrewriteaof is automatically triggered, you can see the server log as follows:

The process of file rewriting

The file rewriting process is shown in the following figure:

Regarding the process of file rewriting, there are two points that need special attention:

(1) Rewriting is performed by the parent process fork child process;

(2) The write command executed by Redis during rewriting needs to be appended to the new AOF file. For this reason, Redis introduces the aof_rewrite_buf cache.

Compared with the above figure, the process of file rewriting is as follows:

1) The Redis parent process first judges whether there is a child process currently executing bgsave/bgrewriteaof. If so, the bgrewriteaof command returns directly. If there is a bgsave command, wait for the bgsave execution to complete before executing it. As mentioned earlier, this is mainly based on performance considerations.

2) The parent process executes the fork operation to create a child process, and the parent process is blocked during this process.

3.1) After the parent process forks, the bgrewriteaof command returns the message "Background append only file rewrite started" and no longer blocks the parent process, and can respond to other commands. All write commands of Redis are still written into the AOF buffer, and are synchronized to the hard disk according to the appendfsync strategy to ensure the correctness of the original AOF mechanism.

3.2) Since the fork operation uses the copy-on-write technology, the child process can only share the memory data during the fork operation. Since the parent process is still responding to the command, Redis uses the AOF rewrite buffer (aof_rewrite_buf in the figure) to save this part of the data to prevent the loss of this part of the data during the generation of the new AOF file. That is to say, during the execution of bgrewriteaof, Redis write commands are simultaneously appended to two buffers, aof_buf and aof_rewrite_buf.

4) The child process writes to a new AOF file according to the command merge rules according to the memory snapshot.

5.1) After the child process writes the new AOF file, it sends a signal to the parent process, and the parent process updates the statistical information, which can be viewed through info persistence.

5.2) The parent process writes the data in the AOF rewrite buffer to the new AOF file, thus ensuring that the database state saved in the new AOF file is consistent with the current state of the server.

5.3) Replace the old file with the new AOF file to complete AOF rewriting.

3. Load at startup

As mentioned earlier, when AOF is turned on, Redis will first load the AOF file to restore data when it starts; only when AOF is turned off, it will load the RDB file to restore data.

When AOF is enabled and the AOF file exists, Redis starts the log:

When AOF is enabled, but the AOF file does not exist, it will not be loaded even if the RDB file exists (some earlier versions may load, but 3.0 will not), and the Redis startup log is as follows:

file verification

Similar to loading an RDB file, when Redis loads an AOF file, it will verify the AOF file. If the file is damaged, an error will be printed in the log, and Redis will fail to start. However, if the end of the AOF file is incomplete (sudden machine downtime, etc. may easily cause the end of the file to be incomplete), and the aof-load-truncated parameter is enabled, a warning will be output in the log, Redis ignores the end of the AOF file, and the startup is successful. The aof-load-truncated parameter is enabled by default:

fake client

Because Redis commands can only be executed in the client context, and when loading the AOF file, the command is read directly from the file, not sent by the client; therefore, before the Redis server loads the AOF file, it will create a The client connected to the network, and then use it to execute the commands in the AOF file, the effect of command execution is exactly the same as that of the client with network connection.

4. AOF common configuration summary

The following are commonly used configuration items for AOF, as well as their default values; those introduced earlier will not be described in detail here.

• appendonly no: Whether to enable AOF
• appendfilename "appendonly.aof": AOF filename
• dir ./: the directory where RDB files and AOF files are located
• appendfsync everysec: fsync persistence strategy
• no-appendfsync-on-rewrite no: Whether to disable fsync during AOF rewriting; if this option is enabled, the load on the CPU and hard disk (especially the hard disk) can be reduced when the file is rewritten, but the data during AOF rewriting may be lost; Need to balance load and security
• auto-aof-rewrite-percentage 100: One of the trigger conditions for file rewrite
• auto-aof-rewrite-min-size 64mb: file rewrite triggers one of the commits
• aof-load-truncated yes: If the end of the AOF file is damaged, whether to still load the AOF file when Redis starts

5. Program selection and common problems

The details of the RDB and AOF persistence schemes are introduced above. The following introduces the characteristics of RDB and AOF, how to choose a persistence scheme, and the problems often encountered in the persistence process.

1. Advantages and disadvantages of RDB and AOF

RDB and AOF have their own advantages and disadvantages:

RDB persistence

Advantages: RDB files are compact, small in size, fast in network transmission, suitable for full copy; recovery speed is much faster than AOF. Of course, one of the most important advantages of RDB compared to AOF is the relatively small impact on performance.

Disadvantages: The fatal disadvantage of RDB files is that the persistence method of data snapshots determines that real-time persistence cannot be achieved. Today, when data is becoming more and more important, a large amount of data loss is often unacceptable, so AOF persistence become mainstream. In addition, RDB files need to meet a specific format and have poor compatibility (for example, old versions of Redis are not compatible with new versions of RDB files).

AOF persistence

Corresponding to RDB persistence, the advantage of AOF is that it supports second-level persistence and good compatibility. The disadvantage is that the file is large, the recovery speed is slow, and it has a great impact on performance.

2. Persistence strategy selection

Before introducing the persistence strategy, we must first understand that whether it is RDB or AOF, the opening of persistence has to pay a performance price: for RDB persistence, on the one hand, the Redis main process will be blocked when bgsave performs the fork operation, and on the other hand On the one hand, writing data to the hard disk by the child process will also bring IO pressure; for AOF persistence, the frequency of writing data to the hard disk is greatly increased (second level under the everysec strategy), the IO pressure is greater, and may even cause AOF additional blocking problems (This blocking will be described in detail later). In addition, the rewriting of AOF files is similar to RDB's bgsave, and there will be blocking during fork and IO pressure problems of sub-processes. Relatively speaking, since AOF writes data to the hard disk more frequently, it will have a greater impact on the performance of the Redis main process.

In the actual production environment, there will be a variety of persistence strategies according to different situations such as data volume, application security requirements for data, and budget constraints; such as not using any persistence at all, using RDB or AOF, Or enable RDB and AOF persistence at the same time. In addition, the choice of persistence must be considered together with Redis's master-slave strategy, because master-slave replication and persistence also have the function of data backup, and the host master and slave slave can independently choose the persistence scheme.

The following discusses the selection of persistence strategies in different scenarios. The following discussion is only for reference, and the actual solution may be more complex and diverse.

(1) It doesn't matter if the data in Redis is completely discarded (for example, Redis is completely used as a cache for DB layer data), then no persistence is required whether it is a stand-alone machine or a master-slave architecture.

(2) In a stand-alone environment (for individual developers, this situation may be more common), if ten minutes or more of data loss can be accepted, choosing RDB is more beneficial to the performance of Redis; if only second-level data loss is acceptable If data is lost, AOF should be selected.

(3) But in most cases, we will configure the master-slave environment. The existence of slaves can not only realize hot backup of data, but also perform read-write separation to share Redis read requests, and continue to provide services after the master crashes.

In this case, one possible approach is to:

master: Completely turn off persistence (including RDB and AOF), so that the performance of the master can reach the best

slave: Close RDB, open AOF (if the data security requirements are not high, you can also open RDB and close AOF), and regularly back up persistent files (such as backing up to other folders, and mark the backup time); then close The automatic rewriting of AOF, and then add a scheduled task, and call bgrewriteaof when Redis is idle every day (such as 12 o'clock in the morning).

Here we need to explain why master-slave replication can be enabled to achieve hot backup of data, and persistence needs to be set? Because in some special cases, master-slave replication is still not enough to ensure data security, for example:

• The master and slave processes stop at the same time: Consider such a scenario, if the master and slave are in the same building or the same computer room, a power outage may cause the master and slave machines to shut down at the same time, and the Redis process stops; if there is no persistence, then Faced with a complete loss of data.
• The master restarts by mistake: Consider a scenario where the master service is down due to a failure. If there is an automatic pull-up mechanism in the system (that is, restart the service after detecting that the service stops), the master will automatically restart. Since there is no persistent file, then the master After restarting, the data is empty, and the slave synchronization data also becomes empty; if neither the master nor the slave is persisted, the data will also be completely lost. It should be noted that even if the sentry is used (the article will introduce the sentry later) for automatic master-slave switching, it may be restarted by the automatic pull-up mechanism before the sentinel polls the master. Therefore, try to avoid "automatic pull-up mechanism" and "no persistence" at the same time.

(4) Off-site disaster recovery: The persistence strategies discussed above are all aimed at general system failures, such as abnormal process exit, downtime, power failure, etc. These failures will not damage the hard disk. However, for some disasters that may cause hard disk damage, such as fire and earthquake, remote disaster recovery is required. For example, in the case of a single machine, you can regularly copy the RDB file or the rewritten AOF file to a remote machine through scp, such as Alibaba Cloud, AWS, etc.; in the case of a master-slave, you can regularly execute bgsave on the master, and then save the Copy the file to the remote machine, or copy the AOF file to the remote machine after executing bgrewriteaof on the slave to rewrite the AOF file. Generally speaking, RDB files are commonly used for disaster recovery due to their small size and fast recovery; the frequency of remote backups is determined according to data security needs and other conditions, but it is best not to be less than once a day.

3. Fork blocking: CPU blocking

In the practice of Redis, many factors limit the memory size of a Redis stand-alone machine, such as:

• In the face of a surge in requests, when it is necessary to expand the capacity from the library, the Redis memory is too large and the expansion time is too long;
• When the host is down, the slave library needs to be mounted after the host is switched, and the Redis memory is too large to cause the mount speed to be too slow;
• And the fork operation in the persistence process is described in detail below.

First explain the fork operation:

The parent process can create a child process through the fork operation; after the child process is created, the parent and child processes share the code segment and do not share the data space of the process, but the child process will obtain a copy of the data space of the parent process. In the actual implementation of the operating system fork, the copy-on-write technology is basically adopted, that is, before the parent/child process tries to modify the data space, the parent and child processes actually share the data space; but when any of the parent/child processes try to modify the data space When there is space, the operating system makes a copy of the modified portion (a page of memory).

Although when fork, the child process will not copy the data space of the parent process, but will copy the memory page table (the page table is equivalent to the index and directory of the memory); the larger the data space of the parent process, the larger the memory page table, and copy it when fork It will also take more time.

In Redis, whether it is RDB persistent bgsave or AOF rewritten bgrewriteaof, it is necessary to fork a child process to operate. If the Redis memory is too large, it will take too much time to copy the memory page table during the fork operation; while the Redis main process is completely blocked when the fork is performed, which means that it cannot respond to the client's request, which will cause the request to be delayed too much .

For different hardware and different operating systems, the time consumption of the fork operation will vary. Generally speaking, if the Redis stand-alone memory reaches 10GB, the time consumption of the fork may reach the level of 100 milliseconds (if the Xen virtual machine is used, this time-consuming may reach the second level). Therefore, generally speaking, the memory of a Redis stand-alone machine should be limited to less than 10GB; however, this data is not absolute, and can be adjusted by observing the time-consuming of fork in the online environment. The observation method is as follows: Execute the command info stats to view the value of latest_fork_usec in microseconds.

In order to alleviate the blocking problem caused by the fork operation, in addition to controlling the size of the Redis stand-alone memory, you can also moderately relax the trigger conditions for AOF rewriting, choose physical machines or virtualization technologies that efficiently support fork operations, such as using Vmware or KVM virtualization machine, do not use a Xen virtual machine.

4. AOF additional blocking: hard disk blocking

As mentioned earlier, in AOF, if the file synchronization policy of the AOF buffer is everysec, then: in the main thread, the system write operation is invoked after the command is written into aof_buf, and the main thread returns after the write is completed; the fsync synchronization file operation is performed by a dedicated The file synchronization thread is called every second.

The problem with this approach is that if the hard disk load is too high, the fsync operation may take longer than 1s; if the Redis main thread continues to write commands to aof_buf at high speed, the hard disk load may increase and the IO resource consumption will be faster; if At this time, the Redis process exits abnormally, and more and more data will be lost, which may far exceed 1s.

For this reason, Redis's processing strategy is as follows: each time the main thread performs AOF, it will compare the time of the last successful fsync; if it is less than 2s from the last time, the main thread returns directly; if it exceeds 2s, the main thread blocks until fsync is synchronized Finish. Therefore, if the system hard disk load is too heavy and the fsync speed is too slow, it will cause the Redis main thread to be blocked; in addition, with the everysec configuration, AOF may lose up to 2s of data instead of 1s.

The method of AOF additional blocking problem location:

(1) Monitor aof_delayed_fsync in info Persistence: When AOF additional blocking occurs (that is, the main thread is blocked while waiting for fsync), this indicator is accumulated.

(2) Redis log when AOF is blocked:

Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

(3) If AOF additional blocking occurs frequently, it means that the hard disk load of the system is too large; you can consider replacing the hard disk with faster IO speed, or analyze the IO load of the system through IO monitoring and analysis tools, such as iostat (system-level io), iotop (io version of top), pidstat, etc.

5. info command and persistence

Some methods of viewing persistence-related status through the info command were mentioned earlier, which are summarized below.

（1）info Persistence

The execution results are as follows:

The more important ones include:

• rdb_last_bgsave_status: The result of the last bgsave execution, which can be used to find bgsave errors
• rdb_last_bgsave_time_sec: last bgsave execution time (unit is s), can be used to find out whether bgsave takes too long
• aof_enabled: whether AOF is enabled
• aof_last_rewrite_time_sec: The execution time of the last file rewrite (unit is s), which can be used to find out whether the file rewrite takes too long
• aof_last_bgrewrite_status: The result of the last bgrewrite execution, which can be used to find bgrewrite errors
• aof_buffer_length and aof_rewrite_buffer_length: aof buffer size and aof rewrite buffer size
• aof_delayed_fsync: AOF additional blocking statistics

（2）info stats

Among them, the one that has a greater relationship with persistence is: latest_fork_usec, which represents the time spent on the last fork, see the previous discussion.

6. Summary

The main content of this article can be summarized as follows:

1. The role of persistence in Redis high availability: data backup, compared with master-slave replication, emphasizes backup from memory to hard disk.

2. RDB Persistence: Backup data snapshots to the hard disk; introduce its trigger conditions (including manual triggering and automatic triggering), execution process, RDB files, etc. It is particularly important to note that the file save operation is performed by a fork child process .

3. AOF Persistence: Back up the executed write commands to the hard disk (similar to MySQL’s binlog), introduce its opening method, execution process, etc., and pay special attention to the selection of file synchronization strategy (everysec), file rewriting process.

4. Some practical issues: including how to choose a persistence strategy, as well as fork blocking, AOF additional blocking, etc. that need attention.

references

"Redis Development and Operation and Maintenance"

"Redis Design and Implementation"

"Redis in Action"

REDIS persistence -- Redis Chinese User Group (CRUG)