Some pits in [Redis] (3) - "Data Persistence"

Redis provides two data persistence methods, RDB and AOF, where RDB is a data snapshot, and AOF records each write command to the log file.

Previous review:

• Some pits in [Redis] (1) - "Common Commands"
  
• Some pits in [Redis] (2) - "Master-Slave Replication" article
  

1 Master downtime causes Slave data loss

Data loss issues can occur if Redis is deployed in the following pattern:

• Master-slave + Sentinel deployment;
• Master does not enable data persistence;
• The Redis process is managed by supervisor and configured as "process down, automatic restart".

If the Master instance is down at this time, the following problems will occur:

• The Master is down and the sentinel has not yet initiated the switch. At this time, the Master process is automatically pulled up by the supervisor immediately;
• However, since the Master does not enable any data persistence, it is an "empty" instance after startup;
• At this point, in order to be consistent with the Master, the Slave will automatically "clear" all data in the instance.

2 AOF's everysec blocks the main thread

When using Redis' AOF, you need to configure the AOF refresh strategy . Based on the balance between performance and data security , the appendfsync ****everysec scheme will be used. That is, the background thread of Redis flushes the data of the AOF page cache to the disk (fsync) every 1 second .

The advantage of this is that the time-consuming operation of AOF brushing can be executed in the background thread, avoiding the impact on the main thread.

But does this scheme really not affect the main thread?

the answer is negative. Imagine a scenario: when the Redis background thread is executing the AOF page cache flush (fysnc), if the disk IO load is too high, the call to fsync will be blocked. However, the main thread at this time still receives write requests, so the main thread at this time will first judge whether the background thread has been successfully flushed last time.

How does Redis judge whether it has been refreshed?

After the background thread is successfully flushed, the flush time will be recorded. The main thread will judge based on this time how long it has been since the last time the disk was refreshed. The whole process is like this:

1) Before the main thread writes the AOF page cache (write system call), it first checks whether the background fsync has been completed;

2) fsync has been completed, the main thread directly writes the AOF page cache;

3) If fsync is not completed, check how long it has been since the last fsync:

4) If the last successful fysnc is within 2 seconds, the main thread will return directly without writing the AOF page cache

5) If more than 2 seconds have passed since the last successful fysnc, the main thread will force the AOF page cache to be written (write system call)

6) Because the disk IO load is too high, at this time, the background thread fynsc will block, and the main thread will also block waiting when writing the AOF page cache ( operating the same fd, fsync and write are mutually exclusive, one party must Wait for the other party to succeed before continuing to execute, otherwise block and wait )

From the above analysis, it can be found that even if the configured AOF flushing strategy is appendfsync everysec, there is still a risk of blocking the main thread: the high disk IO load causes fsync to block, which in turn causes the main thread to block when writing to the AOF page cache .

Therefore, it is necessary to ensure that the disk has sufficient IO resources to avoid this problem.

3 AOF everysec not only lose 1 second of data

I continue to analyze the above problem. When the above step 4) is executed, that is, when the main thread is writing the AOF page cache, it will first judge the time of the last successful fsync. If the last fysnc succeeded within 2 seconds , then the main thread will return directly without writing the AOF page cache.

This means that when the background thread executes the fsync flush, the main thread will not write the AOF page cache until it waits for at most 2 seconds. If Redis crashes at this time, then the AOF file will lose 2 seconds of data instead of 1 second.

Why does the Redis main thread wait for 2 seconds without writing the AOF page cache?

In fact, when **Redis AOF is configured as appendfsync everysec, in theory, the background thread will perform fsync flushing every 1 second. If the disk resources are sufficient, it will not be blocked. ** However, the author of Redis considered that if the disk IO resources are relatively tight at this time, then the background thread fsync has a probability of blocking risk. Therefore, before the main thread writes the AOF page cache, the author first checks the time since the last successful fsync. If it is more than 1 second without success, then the main thread can know at this time that fsync may be blocked.

Therefore, the main thread will wait for 2 seconds without writing the AOF page cache. The purpose is to:

• Reduce the risk of blocking the main thread (if the AOF page cache is written mindlessly, the main thread will block immediately)
• If fsync blocks, the main thread will give the background thread 1 second to wait for the fsync to succeed

This solution should be a further trade-off for Redis authors between performance and data security. Therefore , even if AOF is configured to refresh the disk every second, when the above extreme situation occurs, the data lost by AOF is actually 2 seconds .

4 OOM when RDB and AOF rewrite

When Redis does RDB snapshot and AOF log rewriting, it will create sub-processes to persist the data in the instance to disk. When the child process is created, the forkfunction . After the execution is completed, the parent process and the child process will share the same memory data at the same time.

However, the main process at this time can still receive write requests, and the incoming write requests will use the Copy On Write method to operate memory data.

If the parent process wants to modify a key, it needs to copy the original memory data to the new memory. This process involves the application of "new memory" .

If the business characteristic is "write more and read less" and the OPS is very high, then during RDB and AOF rewrite, a lot of memory copy work will be generated.

If the machine memory resources are insufficient, this will lead to the risk of Redis being OOM!

so:

In order to prevent Redis OOM during RDB and AOF rewrite, memory should be reserved for the Redis machine