Redis is oriented "key-value" type distributed database system NoSQL data, with high performance, persistent storage, adaptation advantage of high concurrent application scenarios like. Although it started late, but the development is very rapid.
Recently, Redis authors wrote in the blog, he saw all Redis for discussion, the persistent misunderstanding of Redis is the biggest, so he wrote a long article on Redis persistence to carry out a systematic discussion. This paper mainly includes three aspects: Redis persistence is how it works, the performance and reliability as well as other types of database comparison. The following is the content of the article:
a, Redis persistence How does it work?
What is persistence? Simply put, that is, after the data into the device off data is not lost, that is our usual understanding of the hard disk. First we look at the database writes are carried out in the end what had been done thing, there are the following five processes:
- The client sends to the server writes (data in memory client).
- Database server receives data (data in memory in the server side) write requests.
- The server calls the write system call, the data is written to disk (data in system memory buffer).
- Operating system to transfer data in the buffer to the disk controller (data in the disk cache).
- Disk controller writes data to the physical disk media (data is actually falls on disk).
Failure Analysis
write roughly the top 5 processes, we combine the following top five procedures a look at the various levels of failure:
- When the database system failure, this time the system kernel is still intact. So this time as long as we finished the implementation of step 3, then the data is safe, because the operating system will follow to complete a few steps behind, to ensure that the data on the disk will eventually fall.
- When the system is powered down, this time in the above-mentioned five of all caches will be disabled, and the database and operating system will stop working. Therefore, only when the data after the completion of step 5, in order to ensure data is not lost after a power outage.
By understanding the 5 steps above, we may want to find out some of the following questions:
- How long database called once write, write data to the kernel buffer?
- How long will the system kernel data buffer is written to the disk controller?
- Disk controller and at what time the data is written to the cache on physical media?
For the first question, the database level will usually full control. And the second question, the operating system has its default policy, but we can also provide a series by POSIX API fsync command forces the operating system to write data from the core area on the disk controller. For the third question, as if the database has been unable to reach, but in fact, in most cases the disk cache is set off, or open only cache is a read cache, that does not write directly to disk . The recommended practice is only turned on when you write cache when the disk devices have battery backup.
Data corruption
so-called data corruption, data can not be recovered is, above all we are talking about how to ensure that the data is actually written to disk up, but that does not mean the data may be written on the disk will not be damaged. For example, we might write a request will be two different write operation, when an accident may result in a write operation to complete safety, but another has not been carried out. If the data file structure organization of the database unreasonable, might result in data completely unrecoverable situation occurs.
Here there are usually three strategies to organize data, to prevent damage to data files can not be restored in case:
- The first processing is the roughest, is no guarantee recoverability of data in the form of organizational data. Instead, after the data file is damaged to recover the data backed up by the backup data synchronization arranged. Indeed MongoDB without opening operation logs by configuring Replica Sets is the case.
- Another is to add on the basis of the above operating on a log, remember what operating behavior of each operation, so that we can recover data by operating log. Because the operation log is appended, in order to write the way, so the situation can not be restored operation log does not appear. It also opened a case like MongoDB operation log.
- Safer approach is to modify the database without the old data, but with an additional way to complete a write operation, so that the data itself is a log, so that the data can not be recovered case never appeared. In fact CouchDB is an excellent example of this approach.
1, a Redis persistence strategy: RDB snapshot
Redis supports snapshot of the current data is saved as a persistence mechanism for data files. The database is how to generate a sustained write a snapshot of it. With a copy fork Redis commands on write mechanism. When generating a snapshot, the current process will fork a child process, then loop through all of the data in the child process, the data is written as RDB file.
We can configure the timing RDB snapshots generated by the Redis save command, for example, you can configure when 10 minutes or less 100 writes to take a snapshot, you can configure when there are 1000 written on a snapshot in 1 hour, you can embodiment with a plurality of rules. These rules are defined in Redis configuration file, you can also set rules Redis Redis running through the CONFIG SET command, no need to restart Redis.
Redis of RDB file is not broken, because of its write operation is performed in a new process, when generating a new RDB file, Redis generated child process will first write data to a temporary file, and then by atom of the system call rename the temporary file is renamed to RDB files so fails at any time, Redis of RDB files are always available.
Meanwhile, the RDB file Redis Redis master-slave synchronization is implemented in the inside of a ring.
However, we can clearly see, RDB has its shortcomings, once the database is a problem, then save the file in our RDB data is not new, RDB file generated from the last stop to Redis data from this period all lost. In some businesses, this can be tolerated, we also recommend ways to use these services RDB be persistent, because the price is not high RDB open. But for some additional high data security requirements of the application, the application can not tolerate data loss, RDB can not do anything, so Redis introduces another important persistence mechanism: AOF log.
2, Redis second persistence strategy: AOF log
AOF is the full name of the log Append Only File, from the name we can tell, it is written to a log file append. The difference is that with the general database, AOF file is recognizable plain text, its content is one of the standard Redis commands. For example, we carried out the following experiment, Redis2.6 version, set to open in the startup command AOF function parameters:
- ./redis-server --appendonly yes
We then execute the following command:
- redis 127.0.0.1:6379> set key1 Hello
- OK
- redis 127.0.0.1:6379> append key1 " World!"
- (integer) 12
- Redis 127.0. 0.1: 6379> of key1
- (integer) 1
- redis 127.0.0.1:6379> del non_existing_key
- (integer) 0
Then we look at AOF log file, you will get the following:
- $ cat appendonly.aof
- *2
- $6
- SELECT
- $1
- 0
- *3
- $3
- set
- $4
- key1
- $5
- Hello
- *3
- $6
- append
- $4
- key1
- $7
- World!
- *2
- $3
- of the
- $4
- key1
You can see, the write operation generates a corresponding command as a log. It is worth noting that last a del command, it has not been recorded in the log AOF, because Redis judge this command does not make changes to the current data set. There is no need to record the write command useless. Further AOF log is not entirely according to the client's request to generate a log, such as command INCRBYFLOAT the chronograph AOF logs have been recorded into a SET record, because floating point operations may be different in different systems, so to avoid with a log generate different sets of data on different systems, so here only the result of the recording operation by the sET.
AOF rewrite
you can think of you every single write command generates a log, then the file is not AOF will be great? The answer is yes, AOF documents will become increasingly large, so Redis also provides a feature called AOF rewrite. Its function is to re-generate an AOF document, the new AOF file a record of the operation only once, rather than as an old document, may be recorded in a number of operations on the same values. RDB and its generation process is similar, but also fork a process directly through the data, write new AOF temporary files. In the process of writing a new file, all written or will write the log old AOF original file, while also recorded in the memory buffer. When the operation is complete re-finished, all buffers will be written to the log-time temporary file. Then call the atomic rename command with a new document to replace the old AOF AOF file.
Two, Redis persistence performance is reliable?
We can see from the above process, RDB is the order of IO operations, performance is very high. While at the same time to recover from the RDB database file, the read data sequence is loaded into memory. So it will not cause random disk read errors.
The AOF is a write file operation, its purpose is to operate the log to disk, so it will also say that we encountered above write operation five processes. So write AOF operational safety but also how much of it? In fact, this can be set in the Redis to write after writing AOF call, when to call fsync be written to disk is controlled by appendfsync option, the following three settings appendfsync, the security strength has been strong .
1, appendfsync no
When set appendfsync to no time, Redis will not take the initiative to call fsync AOF log sync to disk, so all of this is completely dependent on the debugging of the operating system. For most Linux operating system, is carried out every 30 seconds fsync, the data buffer is written to disk.
2, appendfsync everysec
when the set time appendfsync to everysec, Redis will fsync default once every second call, the data buffer is written to disk. But when this time fsync call longer than 1 second. Redis will fsync adopted strategy of delay, wait a second. Fsync is carried out in about 2 seconds after, this time fsync will perform no matter how long it will be. This time because the file descriptor will be blocked when the fsync, so the current write operation will be blocked.
So, the conclusion is: In most cases, Redis will be carried out once every second fsync. In the worst case, two seconds will perform an fsync operation.
This operation is referred to in most database systems group commit, it is a combination of multiple data write operation, a one-time write logs to disk.
3, appednfsync always
when set appendfsync is always, every write operation will be called once fsync, then the data is the safest, of course, since every time the implementation of fsync, so its performance will be affected.
For pipelining What is the difference?
For the operation of pipelining, which is a particular client process sends N-time command, and then waits for the command to return the N results are returned together. By using pipilining it would mean giving up the confirmation of the return value of each command. Since in this case, N is the command executed in the same execution. When the set appendfsync to everysec, there may be some deviation, because it is possible to perform N commands for more than 1 seconds or 2 seconds. But we can guarantee that will not exceed the maximum time the execution time and the N command.
Third, the comparison of databases and other
data security operating system level above, we have talked a lot, in fact, different databases in achieving much the same. In short, the final conclusion is that, in the case of Redis open AOF, the stand-alone data security is not weaker than those of mature SQL database.
Comparison of the data in terms of import of
these data persistence What's the use, of course, it is for data recovery after the restart. Redis is an in-memory database, whether or RDB AOF, just to ensure that their data recovery measures. So Redis in the use of RDB and AOF recovery, will read RDB or AOF file, re-loaded into memory. MySQL database and other start-up time is relative, a lot of the president, because MySQL would have been no need to load data into memory.
But relatively speaking, the provision of services after startup MySQL, which is accessed hot data will gradually loaded into memory, we usually call it warm-up, but before the completion of warming up, its performance will not be too high. The advantage of Redis is a one-time load data into memory, the one-time warm-up. So long as Redis startup is complete, then the speed of service is very fast.
And in the use of RDB and start using the AOF, its start time there are some differences. RDB's start-up time will be shorter, there are two reasons, first, RDB file only one record for each piece of data, it will not be as likely to have multiple operations recorded a data log as AOF. So just write data once each on the line. Another reason is the storage format and the data encoding format Redis RDB file in memory is the same, no further data coding. On the CPU consumption is far less than the load AOF log