Redis persistence of principle and optimization

More welcome attention to micro-channel public number: All food engineer Xiaohui ~

Redis automatically periodically provides a persistent data to the hard drive capacity, and AOF RDB including two programs, the two schemes have their respective advantages and shortcomings may be run simultaneously and together, ensure data stability.

RDB

RDB save a snapshot of data to a file for persistence. RDB and Mysql Dump similar operation.

Implementation modalities

• save. Synchronous operation, will block Redis.
• bgsave. Linux call of fork (), then use the new thread to execute the copy. But also blocked during the fork Redis, but the blocking time is usually very short.
• Automatically saved. Redis configuration file set up an automatic trigger mechanism to save, you can custom modifications, the same operating principle bgsave.

And save the comparison bgsave

note:

• If you run multiple Redis on the machine, you need to configure RDB file name, or multiple Redis of RDB file will overwrite each other.

In addition to these three modes, the following will generate RDB files:

• The amount of the whole master-slave replication, host generates RDB file.
• debug reload Redis is to provide debug level restart, restart without emptying one kind of memory, this approach will trigger the generation of RDB files.
• The implementation of shutdown, will trigger the generation of RDB files.

RDB shortcomings

• The full amount of data stored and time-consuming.
• While fork () uses copy-on-write strategy, but still consume memory
• Write RDB files consume a lot of IO performance.

AOF

When using AOF lasting way, Redis will each write request is recorded in a log file, AOF operation and Mysql Binlog similar. By reducing the volume file AOF AOF rewrite mechanism, thereby reducing the recovery time.

Implementation modalities

• always. Redis commands of each write buffer is written to the system, then each write commands use fsync "write" the hard disk.
• everysec. Process is always the same, but fsync frequency of once per second. This is the default configuration Redis, if the system goes down, you lose about one second data
• no. Determined by the operating system when flushed from the system buffer to the hard disk.

AOF rewrite

In order to solve the problem AOF file volume expansion, Redis provides AOF rewrite function: Redis server can create a new document to replace the existing AOF AOF files, database files saved two old and the new state is the same, but the new AOF command file does not contain any redundant waste of space, typically the volume will be much smaller than the old AOF file.

AOF rewrite mode

• bgrewriteaof (similar to process and bgsave)
  
• AOF rewrite the configuration (similar to the auto-save RDB)

AOF AOF rewrite the original file does not need to make any reading, writing, analysis and other operations, this function is read by the current state of the database server to achieve.

RDB vs AOF

Redis data loading at start

Redis start the data loading process:

1. When AOF AOF and there is persistent open file, the file is loaded first AOF.
2. When AOF AOF close or file does not exist, load RDB file.
3. After loading AOF / RDB file successfully, Redis started successfully.
4. AOF / RDB files when there is an error, Redis fails to start and print an error message.

Development of operation and maintenance of the common problems

fork operation

fork () is the actual cost of copying the parent's page tables and create a process descriptor to the child process, so the speed is generally faster

The larger the amount of memory, the longer the time-consuming; relatively fast physical machine, the virtual machine is relatively slow.

Optimization

1. Preferentially used physical machine or fork support efficient operation of virtualization technology
2. Examples of the maximum available memory control Redis maxmemory
3. The rational allocation of Linux memory allocation strategy: vm.overcommit_memory = 1. The default value is 0, while Linux makes memory allocation, memory is not found inadequate, it will not be distributed, and cause obstruction fork
4. Lower fork frequency. Such as relaxing AOF rewrite automatically trigger timing or reduce unnecessary master copy from the whole amount

Cost-of-process

• CPU. RDB and AOF file generation, are CPU-intensive. Do not Redis process bound to a particular CPU, single core to prevent overload; while not Redis and CPU-intensive applications deployed together.
  
• RAM. fork memory overhead, copy-on-write.
  
• hard disk. AOF RDB and write files. Iostat iotop can be combined and analyzed.

Optimization

1. Do the hard drive and high load service deployed together: storage services, message queues, etc.
2. Configuration no-appendfsync-on-rewrite = yes. During this AOF rewrite, not to carry out additional operations AOF (the main thread only writes data buffers), can reduce memory overhead.

But if AOF rewrite period, Redis is down, then the Linux system default configuration, up to the 30s will lose data. If you can not tolerate data loss, no-appendfsync-on-rewrite configuration no; if applications can not tolerate delay, and can tolerate a small amount of data loss, it is set to yes.

1. Decided to write disk type according to the amount: for example ssd
2. Stand-alone multi-instance persistence file directory can be considered sub-plate, or use a similar mechanism to the hard disk cgroups rational allocation of resources

AOF additional obstruction

For example, in everysec policy AOF, the main thread will fsync than the last time, if fsync over two seconds from the last time, it will cause the main thread blocking (synchronous waiting thread synchronization is complete).

Daily developers can use info persistencethe command to check the number of blocked AOF history occurred; however, need to know the time of occurrence of the AOF additional blocking Redis need to view the log.

Send AOF additional blocking when the log is as follows:

Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.

Optimization method (refer to other aspects of the optimization points)

More welcome attention to micro-channel public number: All food engineer Xiaohui ~