Redis supports data backup, that is, data backup in master-slave mode.
Redis supports data persistence, which can keep the data in memory in the disk, and can be loaded again for use when restarting.
Although Redis is a memory-based storage system, it itself supports the persistence of in-memory data, and provides two main persistence strategies: RDB snapshots and AOF logs. Memcached does not support data persistence operations.
1) RDB snapshot
Redis supports a persistence mechanism that saves a snapshot of the current data as a data file, that is, an RDB snapshot. But how does a continuously writing database generate snapshots? Redis relies on the copy on write mechanism of the fork command. When generating a snapshot, fork the current process out of a child process, then loop all the data in the child process, and write the data as an RDB file. We can configure the timing of RDB snapshot generation through the save command of Redis. For example, it can be configured to generate a snapshot in 10 minutes, or it can be configured to generate a snapshot after 1000 writes, or multiple rules can be implemented together. These rules are defined in the Redis configuration file. You can also set the rules when Redis is running through the Redis CONFIG SET command without restarting Redis.
The RDB file of Redis will not be damaged, because its write operation is performed in a new process. When a new RDB file is generated, the subprocess generated by Redis will first write the data to a temporary file, and then use the atomic The rename system call renames a temporary file to an RDB file, so that in the event of a failure, the RDB file for Redis is always available. At the same time, the RDB file of Redis is also a part of the internal implementation of Redis master-slave synchronization. RDB has its shortcomings, that is, once there is a problem with the database, the data saved in our RDB file is not brand new, and all the data from the last RDB file generation to the time when Redis was down is lost. Under some businesses, this is tolerable.
Advantages of RDBs
- An RDB is a very compact file that holds a Redis dataset at a point in time. This kind of file is great for backups: for example, you can back up an RDB file every hour for the last 24 hours, and an RDB file every day of the month. This way, you can always restore your dataset to a different version, even if you run into problems.
- RDB is great for disaster recovery: it's a single file, and it's so compact that it can be sent (after encryption) to another data center, or to Amazon S3.
- RDB can maximize the performance of Redis: the only thing the parent process has to do when saving the RDB file is to fork a child process, and then the child process will handle all the next save work, and the parent process does not need to perform any disk I/O operations .
- RDB is faster than AOF when recovering large datasets.
Disadvantages of RDBs
- If you need to try to avoid losing data in the event of a server failure, then RDB is not for you. Although Redis allows you to set different save points to control how often the RDB file is saved, it is not an easy operation because the RDB file needs to save the state of the entire dataset. So you may save RDB files at least once every 5 minutes. In this case, in the event of an outage, you could lose several minutes of data.
- Every time the RDB is saved, Redis has to fork() a child process, and the child process does the actual persistence work. When the data set is large, fork() can be very time-consuming, causing the server to stop processing clients within a certain millisecond; if the data set is very large and CPU time is very tight, this stop time may even be long for a full second. Although AOF rewrites also require fork() , no matter how long the interval between executions of AOF rewrites is, there is no loss of data durability.
2) AOF log
The full name of the AOF log is append only file, which is a log file that is appended to. Different from the binlog of the general database, the AOF file is recognizable plain text, and its content is the standard Redis commands one by one. Only those commands that would result in data modification are appended to the AOF file. Each command to modify data generates a log, and the AOF file will become larger and larger, so Redis provides another function called AOF rewrite. Its function is to regenerate an AOF file. The operation of a record in the new AOF file can only be performed once, unlike an old file, which may record multiple operations on the same value. The generation process is similar to RDB. It is also a fork process, which directly traverses the data and writes a new AOF temporary file. In the process of writing a new file, all write operation logs will still be written to the original old AOF file, and will also be recorded in the memory buffer. When the redo operation is completed, the logs in all buffers will be written to the temporary file at one time. Then call the atomic rename command to replace the old AOF file with the new AOF file.
AOF is a file write operation whose purpose is to write the operation log to disk, so it will also encounter the process of the write operation we mentioned above. After calling write to AOF in Redis, the appendfsync option is used to control the time of calling fsync to write it to the disk. The following three setting items of appendfsync gradually increase the security strength.
- appendfsync no When appendfsync is set to no, Redis will not actively call fsync to synchronize the AOF log content to disk, so it all depends on the debugging of the operating system. For most Linux operating systems, an fsync occurs every 30 seconds to write the data in the buffer to disk.
- appendfsync everysec When appendfsync is set to everysec, Redis will make an fsync call every second by default to write the data in the buffer to disk. But when this time the fsync call is longer than 1 second. Redis will adopt a strategy of delaying fsync and wait another second. That is, fsync is performed after two seconds, and this time the fsync will be performed no matter how long it takes to execute. At this time, since the file descriptor will be blocked during fsync, the current write operation will be blocked. So the conclusion is that in the vast majority of cases, Redis will fsync every second. In the worst case, an fsync operation occurs every two seconds. This operation is called group commit in most database systems, which combines the data of multiple write operations and writes the log to disk at one time.
- appendnfsync always When appendfsync is set to always, each write operation will call fsync once. At this time, the data is the safest. Of course, since fsync is executed every time, its performance will also be affected.
Advantages of AOF
- Using AOF persistence makes Redis much more durable: you can set different fsync strategies, such as no fsync , fsync every second , or fsync every time a write command is executed . The default policy of AOF is fsync once per second. Under this configuration, Redis can still maintain good performance, and even if there is a failure, only one second of data will be lost ( fsync will be executed in the background thread, so The main thread can continue to work hard on the command request).
- AOF file is an append only log file, so writing to AOF file does not need to seek , even if the log contains incomplete commands for some reason (such as writing to disk full, write downtime, etc.), the redis-check-aof tool can easily fix this too.
- Redis can automatically rewrite AOF in the background when the size of the AOF file becomes too large: the new AOF file after rewriting contains the minimum set of commands needed to restore the current dataset. The entire rewrite operation is absolutely safe, because Redis will continue to append commands to the existing AOF file during the process of creating a new AOF file. Even if there is a shutdown during the rewriting process, the existing AOF file will not be lost. . Once the new AOF file is created, Redis will switch from the old AOF file to the new AOF file and start appending to the new AOF file.
- The AOF file saves all the write operations performed to the database in an orderly manner. These write operations are saved in the format of the Redis protocol, so the content of the AOF file is very easy to be read by humans, and it is also very easy to parse the file. Exporting AOF files is also very simple: for example, if you accidentally execute the FLUSHALL command, but as long as the AOF file is not overwritten, just stop the server, remove the FLUSHALL command at the end of the AOF file, and restart Redis, The data set can be restored to the state before FLUSHALL was executed.
Disadvantages of AOF
- For the same dataset, the size of AOF files is usually larger than that of RDB files.
- Depending on the fsync strategy used, AOF may be slower than RDB. In general, the performance of fsync per second is still very high, and turning off fsync can make AOF as fast as RDB, even under heavy load. However, when dealing with huge write loads, RDB can provide a more guaranteed maximum latency (latency).
- AOF has had such a bug in the past: due to individual commands, when the AOF file is reloaded, the dataset cannot be restored to the original state when it was saved. (The blocking command BRPOPLPUSH , for example, caused such a bug.) Test suites have added tests for this situation: they automatically generate random, complex data sets and reload them to ensure everything normal. Although this kind of bug is not common in AOF files, it is almost impossible for RDB to have this kind of bug in comparison.
RDB and AOF, which one should I use?
In general, if you want to achieve data security comparable to PostgreSQL, you should use both persistence features.
If you care a lot about your data, but can still tolerate data loss within minutes, then you can just use RDB persistence.
There are many users who only use AOF persistence, but we do not recommend this method: because it is very convenient to generate RDB snapshots regularly for database backup, and the speed of RDB recovery of data sets is faster than that of AOF recovery. Besides, using RDB can avoid the bug of AOF program mentioned earlier.
For the reasons mentioned above, in the future we may integrate AOF and RDB into a single persistence model. (This is a long-term plan.)
Backup Redis data
Before reading this section, keep the following sentence in mind: Make sure to backup your database!
Disk failures, node failures, and the like can make your data disappear, and not backing it up is very dangerous.
Redis is very friendly for data backup, because you can copy RDB files while the server is running: once the RDB file is created, nothing is modified. When the server wants to create a new RDB file, it first saves the contents of the file in a temporary file. When the temporary file is written, the program uses
rename(2)
to atomically replace the original RDB file with the temporary file.
This means that it is absolutely safe to copy RDB files at any time.
Here are our recommendations:
- Create a cron job to back up one RDB file to one folder every hour and one RDB file to another folder every day.
- Make sure that the snapshot backups have the corresponding date and time information, and use the find command to delete expired snapshots each time the periodic task script is executed: for example, you can keep hourly snapshots within the last 48 hours, and you can also keep the most recent snapshots. Daily snapshots for a month or two.
- At least once a day, back up the RDB outside your data center, or at least the physical machine where you run the Redis server.