Affirm
This article was first published from my public account: Yizhihua is not considered romantic . Please indicate the source if reprinted!
Interested friends can follow the personal public number: Yizhihua is not considered romantic
22.jpg
Foreword
Redis is a NO SQL database based on memory, but we all know the data stored in memory. As long as the server is shut down, the data in memory will disappear.
In order to avoid data loss in memory, Redis provides support for persistence. Redis has two persistence mechanisms: RDB and AOF.
You can first look at the principles of two persistence mechanisms:
RDB and AOF.jpg in 01_Redis
Introduction to the two persistence mechanisms of RDB and AOF
RDB persistence mechanism, perform periodic persistence on data in redis
The AOF mechanism uses each write command as a log and writes it to a log file in append-only mode. When redis restarts, the entire data set can be reconstructed by replaying the write instructions in the AOF log
If you use both RDB and AOF persistence mechanisms, then when redis restarts, AOF will be used to rebuild the data, because the data in AOF is more complete
Advantages of RDB persistence mechanism
-
RDB is very suitable for cold backup. You can send this complete data file to some remote secure storage, such as a cloud server.
-
RDB has very little impact on the read and write services provided by redis, which can keep redis high performance, because the main process of redis only needs to fork a child process, and let the child process perform disk IO operations for RDB persistence
-
Compared with the AOF persistence mechanism, it is faster to restart and restore the redis process directly based on RDB data files
Disadvantages of the RDB persistence mechanism
- If you want to lose as little data as possible when redis fails, then RDB is not as good as AOF. Generally speaking, RDB data snapshot files are generated every 5 minutes or longer, at this time you have to accept that once the redis process is down, then the data of the last 5 minutes will be lost
- Every time RDB executes the RDB snapshot data file generation in the fork subprocess, if the data file is particularly large, it may cause the service provided by the client to be suspended for milliseconds, or even seconds.
Advantages of AOF persistence mechanism
- AOF can better protect data from loss. Generally, AOF will perform an fsync operation through a background thread every 1 second, and lose up to 1 second of data.
- AOF log files are written in append-only mode, so there is no overhead of disk addressing, write performance is very high, and the file is not easily damaged, even if the end of the file is damaged, it is easy to repair
- Even if the AOF log file is too large, the background rewrite operation will not affect the client's reading and writing. Because when rewriting the log, the instructions in it will be compressed to create a minimum log that needs to be restored. When creating a new log file, the old log file is written as usual. When the log file after the new merge is ready, exchange the old log file
- AOF log file commands are recorded in a very readable way, this feature is very suitable for emergency recovery of catastrophic accidental deletion. For example, if you accidentally use the flushall command to clear all data, as long as the background rewrite has not occurred at this time, you can immediately copy the AOF file, delete the last flushall command, and then put the AOF file back, you can pass the recovery mechanism, Automatically restore all data
Disadvantages of the AOF persistence mechanism
- For the same data, the AOF log file is usually larger than the RDB data snapshot file
- After AOF is turned on, the supported write QPS will be lower than the write QPS supported by RDB, because AOF is generally configured to fsync log files once per second. Of course, once fsync per second, the performance is still very high.
How to choose RDB and AOF
- Do n’t just use RDB because it will cause a lot of data loss
- Don't just use AOF, because there will be problems. First: cold standby through AOF, no cold standby with RDB, the recovery speed is faster. Second: RDB simply and crudely generates data snapshots every time, which is more robust and can avoid the bugs of AOF, such a complicated backup and recovery mechanism
- Comprehensive use of AOF and RDB two persistence mechanisms, using AOF to ensure that data is not lost, as the first choice for data recovery; using RDB to do different degrees of cold backup, when AOF files are lost or damaged and unavailable, also RDB can be used for fast data recovery
RDB configuration
// 900s内至少达到一条写命令
save 900 1
// 300秒内至少达到10条写命令
save 300 10
// 60秒内至少达到10000条写命令
save 60 10000
You can also manually call the save or bgsave command to perform rdb snapshot generation synchronously or asynchronously
Workflow of RDB persistence mechanism
- Redis tries to generate the RDB snapshot file according to the configuration
- fork a child process out
- The child process attempts to dump the data into a temporary RDB snapshot file
- After the generation of the RDB snapshot file is completed, the old snapshot file is replaced
Every time dump.rdb generates a new snapshot, it will overwrite the old snapshot file
AOF persistent configuration
AOF persistence is off by default, and the RDB persistence configuration is turned on by default.
Configure appendonly yes, you can turn on AOF persistence
After enabling AOF persistence, redis will write to the log file every time it receives a write command. Of course, it will first write to os cache, and then fsync every certain time.
And even if both AOF and RDB are turned on, AOF will be preferred when redis restarts, because AOF data is relatively complete
You can configure AOF's fsync strategy. There are three strategies to choose from:
- always: write a piece of data every time, immediately fsync the write-off of this data to the disk, the performance is very poor
- everysec: fsync the data in os cache to disk every second, this is the most commonly used, and the performance is relatively high
- no: only redis is responsible for writing data to the os cache, no need to manage, rely on os to flash the data into the disk according to its own strategy
AOF rewrite
The data in redis is limited, and many data may automatically expire, may be deleted by users, or may be cleared by redis with a cache clearing algorithm
The data in redis will continue to eliminate the old data, and only part of the commonly used data will be automatically retained in redis memory
So it is very likely that the data that has been cleaned up before, the corresponding write log still stays in the AOF, and the AOF log file will be one, which will continue to swell.
Therefore, based on the above reasons, AOF will automatically do rewrite operations in the background every certain time, for example, the log has been stored for 100w data, and only 10w data in redis memory at this time; rewrite will be based on the current 10w data in memory Build a newest set of logs into AOF, overwriting the old logs
In redis.conf, you can configure the rewrite strategy:
- auto-aof-rewrite-percentage 100
- auto-aof-rewrite-min-size 64mb
If the size exceeds 64mb, and it has increased by 100% from the last time, only one rewrite will be triggered
Specific rewrite steps:
- redis fork a child process
- The child process builds a log based on the data in the current memory and starts writing the log to a new temporary AOF file
- The main process of redis, after receiving the new write operation of the client, writes the log in the memory, and the new log also continues to write to the old AOF file
- Replace old log files with new log files
Repair of broken AOF files
If the machine is down when redis appends data to the AOF file, it may cause damage to the AOF file
Use redis-check-aof --fix command to repair broken AOF file
AOF and RDB work simultaneously
- If the RDB is performing snapshotting operations, redis will not perform AOF rewrite operations. If redis is performing AOF rewrite, then RDB snapshotting will not be performed
- If the RDB is performing snapshotting, and the user executes the BGREWRITEAOF command at this time, the AOF rewrite will not be performed until the RDB snapshot is generated.
- There are both RDB snapshot files and AOF log files, so when redis restarts, AOF will be used for data recovery because the logs are more complete