Redis persistence mode
RDB Snapshot
Outline
By default, Redis database snapshots stored in a binary file named dump.rdb in. You can set the Redis, when it "N seconds dataset has at least M changes" this condition is satisfied, automatically save a data set.
You can also make Redis data sets stored by calling the SAVE or BGSAVE, manually. For example, the following is provided Redis will meet "at least 60 seconds have been changed key 1000" when the condition, automatically save a data set:
save 60 1000
This approach is called persistent snapshots snapshotting.
When a snapshot
The following situations will arise redis snapshot of data in memory
- Users send bgsave command (this time redis will fork a child process, the child process is responsible for generating the hard disk file, the parent is responsible for continuing to accept commands)
- Users send a save command (command and bgsave different, after sending the save command, the system creates the system will no longer receive the new command before the snapshot is complete, in other words save command will block behind the command, but does not bgsave)
- User profile configuration similar such commands save 60 1000
- This meant that, since the last snapshot successful date, if satisfied "There are 1,000 written within 60 seconds," this condition, the system automatically calls bgsave, if there are multiple configuration file save command, only to meet a call bgsave command
- Users send a shutdown, the system will first save command blocking client, and then shut down the server
- When the Lord sent from the server when the primary server architecture from the sync command to perform the copy operation, only the primary server was no bgsave operation, the master server will perform bgsave operation.
Configuration snapshots
save 60 1000
stop-writes-on-bgsave-error no rdbcompression yes
dbfilename dump.rdb
dir ./Way of working
When Redis dump.rdb need to save the file, the server performs the following actions: Redis calls forks have both parent and child processes.
The child process data set is written to a temporary file RDB.
When the child process to complete the writing of a new RDB file, Redis replace the original file with the new RDB RDB file, and delete the old file RDB. This way of working makes Redis can be copied benefit (copy-on-write) from the write mechanism.
Append only file operations (Append-only fi le, AOF)
Outline
The snapshot feature is not very durable (dura ble): If for some reason Redis caused downtime, the server will lose the most recently written, and those data are not yet saved to the snapshot. From version 1.1, Redis adds a completely durable persistent way: AOF persistence.
AOF way you can open the configuration file:
appendonly yes
From now on, whenever a change Redis execute command data sets (such as SET), this command will be appended to the end AOF file. In this case, when Redis restarted, the program can achieve the purpose of reconstruction data sets by re-executing commands in the file AOF.
Log rewrite
Because the mode of operation AOF is continually appended to the end of the command file, so with the increasing write command, the volume of AOF documents will become bigger and bigger. For example, if you call 100 times INCR on a counter, so just to save the current value of the counter, AOF file you need to use 100 record (entry). In practice, however, use the SET command has only one large enough to hold the current value of the counter, and the remaining 99 records are actually redundant.
To handle this situation, Redis supports an interesting feature: you can without interrupting service clients, for AOF documents for reconstruction
(Rebuild). BGREWRITEAOF execute commands, Redis AOF will generate a new file, this file contains the commands required to reconstruct at least the current data set. Redis 2.2 BGREWRITEAOF need to manually execute the command; Redis 2.4, you can automatically trigger AOF rewrite, specific information, please see an example configuration file 2.4.
How durable AOF
How long before you can configure Redis will fsync data to disk first. There are three ways:
Each time a new command is added to the AOF document on the implementation of a fsync: very slow, very safe
fsync once per second: fast enough (and almost persistent use RDB), and only one second of data loss in case of failure. Never fsync: the data to the operating system to handle. Faster and less safe choice.
Recommended (and also the default) measures to fsync once per second, this strategy can take into account the fsync speed and security.
If the file is corrupted AOF how to do?
The server may shut down when the program is being AOF file for writing, if downtime caused AOF file error (corrupt), then restart Redis in AOF will refuse to load this file, ensuring consistency of the data will not be destroyed. When this happens, you can use the following methods to repair the mistakes of AOF documents:
- Create a backup for the existing AOF file.
- Using the supplied Redis of redis-check-aof procedures, the original file AOF repair: redis-check-aof -fi x
- (Optional) using di ff -u contrast repaired file backup AOF AOF and the original file to view the differences between two files.
- Redis server restart, wait AOF file server load repair and data recovery.
AOP rewrite principle
AOF rewrite and RDB create a snapshot, as it is the clever use of copy-on-write mechanism: redis execute fork (), and now have both parent and child processes.
The contents of the new child process began AOF file written to a temporary file.
For all new write command execution, the parent process while they accumulate into a memory cache, while these changes will append to the existing file AOF, even in this kind of rewriting halfway down the existing file AOF still safe.
When the child process is complete rewrite work, it sends a signal to the parent process, the parent process after receiving the signal, all of the additional data in cache memory to the end of the new AOF file.
Get! Now Redis atomically replace the old file with a new file, after all of the commands will be added directly to the end of the new AOF file.
How to switch from RDB way to AOF way
2.2 or later Redis, can restart without switching from RDB to AOF: create a backup file for the latest dump.rdb.
The backup in a safe place.
Execute the following two commands:
-redis-cli config set appendonly yes redis-cli config set save “”
Ensure that the write command is correctly appended to the end AOF file.
The first command opens up AOF features: Redis will block until the file is created until the completion of the initial AOF, after Redis will continue to process the request command and starts a write command appended to the end AOF file.
The second command execution function for closing RDB. This step is optional, if you prefer, you can also use the RDB and AOF both persistence functions simultaneously.
Important: Do not forget to open the AOF function in redis.conf in! Otherwise, after a server restart, before it will be forgotten by configuring CONFIG SET settings, the program to start the server according to the original configuration.
AOF and the interaction between the RDB
In the version number of 2.4 or greater Redis, the process BGSAVE execution, may not be performed BGREWRITEAOF. Conversely, in the course of BGREWRITEAOF execution, you can not perform BGSAVE. This prevents two Redis background processes at the same time a lot of disk I / O operations.
If BGSAVE is being performed, and the user explicitly invoke BGREWRITEAOF command, the server will respond to a user OK status, and inform the user, BGREWRITEAOF has been scheduled to be executed: BGSAVE Once finished, BGREWRITEAOF will officially begin. When Redis start, if RDB AOF persistence and persistence have been opened, the program will give priority to the use of AOF file to restore the data set because AOF files saved data is usually the most complete.
AOF VS RDB
RDB persistent manner can be spaced able to store a snapshot of your data at a specified time.
AOF persistent recorded each write operation to the server when the server restart will re-execute these commands to restore the original data, AOF command to save an additional protocol redis each write operation to the end of the file .Redis also on the AOF files for background rewritten so that AOF
The file size will not be too large.
If you just want your data exists at the time the server is running, you can not use any persistent way.
You can also open two kinds of persistence simultaneously, in this case, when redis restart priority load AOF file to restore the original data, as stored under normal circumstances AOF file data set than the RDB file is saved to complete dataset.
The most important thing is to understand the RDB and AOF persistence of different ways, let us RDB persistent way to start:
RDB advantages and disadvantages
advantage
RDB is a very compact file, which holds some point in time was the data set, ideal for backup data sets, such as what you can save data in the past 24 hours in each of the Times, while preserving the past 30 days every day data, so that even if a problem you can restore to a different version of the data set according to demand.
RDB is a compact, single file, easily transferred to another remote data center or Amazon's S3 (may be encrypted), ideal for disaster recovery.
RDB RDB file when you save the only parent needs to do is fork a child process, then all the work done by the child, the parent is not required to do other IO operations, so RDB persistent way to maximize the performance of redis .
Compared with the AOF, when restoring large data sets, RDB way faster.
Shortcoming
The case if you want to stop working redis accident (eg power failure) of the missing data to say the least, the RDB is not for you. Although you can configure different save time points (for example, every five minutes and the data set has 100 write operation) is complete Redis to save the entire data set is a relatively heavy work, you will usually every 5 minutes longer or do a full save, in the event of unplanned downtime Redis, you may lose a few data minute.
RDB requires frequent fork child process to save the data set to the hard disk, when the data set is relatively large when, fork process is very time-consuming, may lead to Redis within a few milliseconds can not respond to client requests. Huge if the data set and lower CPU performance is not a good situation, this situation will last one second, AOF also need to fork, but the frequency rewrite the log file that you can adjust to improve the durability of the data set.
AOF advantages and disadvantages
Benefits -
AOF will make your use more durable Redis: You can use different strategies fsync: No fsync, fsync per second, each time to write using the default fsync fsync strategy per second, Redis performance is still very good (fsync is made. background thread for processing, the main thread will try to handle client requests), event of a failure, you lose a maximum of 1 second of data.
AOF file is a log file only additional, so do not write seek, even if for some reason (disk space is full, the process of writing down, etc.) is not a complete write command, you can also use redis-check-aof tools to fix these problems.
When Redis AOF may become too large file size, automatically in the background rewriting of AOF: AOF new file after overwriting command contains a minimal set of data required to restore the current collection. Rewrite the entire operation is absolutely safe, because in the process of creating a new Redis AOF file, the command will continue to append to an existing file AOF inside, even during downtime rewrite existing AOF files will not be lost . Once the new AOF file is created, Redis will switch from the old to the new file AOF AOF file and start the new AOF file append operation.
AOF save the file in an orderly manner all write operations performed on the database, the write operation to save the format Redis protocol, so the contents of AOF file is very easy to read people, to file for analysis (parse) is also very easy. Export (export) AOF file is very simple: For example, if you are not careful FLUSHALL execute the command, but as long as AOF file has not been overwritten, so long as the server stops, remove FLUSHALL command AOF end of the file, and restart Redis, you can restore the data set to the state before FLUSHALL execution.
Cons -
For the same data sets, files generally AOF volume greater than the volume RDB file.
According to fsync strategy being used, AOF speed may be slower than RDB. In general, fsync per second performance is still very high, and close fsync allows AOF speed and RDB as fast, even under high loads. But when dealing with a huge write load, RDB can provide the maximum delay time (latency) is more assured.
Redis data backup
In reading this little pre-holiday, keep in mind the following sentence: Make sure your data disk failure, a complete backup node failure by the sort of problems can make your data disappear, no backup is very dangerous.
Redis for data backup is very friendly, because you can make copies of RDB file while the server is running: Once RDB file is created, it will not make any changes. When the server you want to create a new RDB file, it first content files stored in a temporary file which, when complete write temporary files, use the program only rename (2) atomically replace the original file with the temporary file RDB.
This means that whenever a file copy RDB is absolutely safe.
Create a recurring task (cron job), per hour back up a file to a folder RDB, RDB and a daily backup files to another folder.
Snapshot backups are sure that with the appropriate date and time information every time you perform periodic task script, using the fi nd command to delete outdated snapshot: For example, you can keep hourly snapshots within the last 48 hours, you can keep the last daily snapshots of one or two months.
At least once a day, RDB will back up your data center to the outside, or at least back up to beyond the physical machine you run Redis server.
#! /bin/bash
PATH=/usr/local/bin:$PATH
redis-cli SAVE
date=$(date +"%Y%m%d")
cp /var/lib/redis/6379/dump.rdb /data01/cache_backup/$date.rdb
echo "done!"Disaster Recovery
Redis basically disaster recovery data backup and transfer them to a plurality of different external backup data center. Disaster Recovery can be
When a serious problem Redis running and generating a snapshot of primary data center, so that the data is still in a safe state.
Redis because many users are entrepreneurs, they do not have a lot a lot of money to waste, so the introduction of the following are some of the practical and inexpensive disaster recovery backup methods:
Amazon S3, S3 and other similar services, is a good place to build a disaster recovery system. The easiest way is to RDB your hourly or daily backups encrypted and transmitted to S3. Encryption of data can be done (symmetric encryption mode) gpg -c command. Remember to put your password into several different, safe place to go (for example, you can copy the password to your organization's most important person). Use multiple storage services to store data files, you can enhance data security.
Snapshots can be transmitted using the complete SCP (SSH of assembly). The following is a simple and secure delivery method: buy a data center near you very far VPS, installed SSH, create a password-free SSH client key, and add the key to the VPS authorized_keys file, so that you can this snapshot backup file transfer to the VPS. In order to achieve the best data security, at least from two different providers, where each purchase a VPS for data disaster recovery.
It should be noted that this type of disaster recovery system if not carefully handled, it is very easy to fail. Whether the volume and the original volume snapshot file At a minimum, you should be after the file transfer is complete, check the backup file transfer is the same. If you are using a VPS, you can also check whether the transfer complete by comparing SHA1 files and documents to confirm.
In addition, you also need a separate alarm system and let it tell you when the conveyor (transfer) is responsible for sending a backup failures.
