Redis Advanced: Transaction + persistence + optimization

Brief introduction

Redis is an open source (BSD license) is stored in memory data structures, as a database, cache and message broker. It supports such as strings, hash ranked set, a list, set with a range query, bitmaps, hyperloglog, with the radius of the query and the flow index geospatial data structure.

Redis has a built-in replication, Lua script, LRU cleared affairs and different levels of persistent disk, and provides high availability by automatically partitioning and Redis Redis Sentinel cluster.

Affairs

In order to ensure atomicity combination of multiple commands, Redis provides a simple transactional capabilities and integrated Lua script to solve this problem. Brief use of Redis transaction as well as its limitations.

Redis provides a simple transactional capabilities, a set of commands to be executed with multi placed between two and exec command.

• multi commands represent transaction began .
• exec commands represent transaction ends .
• Command is executed therebetween atomic order.
• It Rollback transaction is not supported .

Publish and subscribe

Redis provides a messaging mechanism based on the "publish / subscribe" model, under this model, news publishers and subscribers do not communicate directly, publisher client publish messages to a specified channel (channel), each subscribe to the channel the client can receive the message.

make an announcement

publish channel:sports "Tim won the championship

Subscribe to news

subscribe channel:sports

There are two things about the subscription command attention :

• After executing client subscription command to enter the subscription status can only receive subscribe,

psubscribe, unsubscribe, punsubscribe four command.

• The new open subscription client can not receive messages prior to the channel, because Redis news release will not be persistent.

Redis publish and subscribe to compare the mature MQ

1. MQ support multiple message protocols , including AMQP, MQTT, Stomp et al., And supports JMS specification, but does not provide support for these Redis protocols;
2. MQ provides persistence function , but not Redis persistent storage of messages, once the message is sent, if there are no subscribers to receive, the message will be lost;
3. MQ provides messaging security, when a client connection timeout or transaction rollback from happening, the message will be sent back to the client , Redis did not provide messaging protection.

In short, the function provided by MQ than Redis publish-subscribe complicated, after all, do not specifically Redis publish-subscribe, but if you already have Redis system and requires basic publish-subscribe functionality, there is no need to install MQ, because MQ might provide functionality most are less than, and you can tolerate the shortcomings redis publication subscriptions, you can consider using it.

Endurance of

Redis supports RDB and AOF two kinds of persistence mechanisms , persistence function effectively protect against data loss caused by the process of withdrawal problems when the next restart persistent file before you can use data recovery.

RDB persistence

RDB persistence is a snapshot of the current process data is saved to the hard disk of the process, trigger RDB persistence process is divided into manual and automatic trigger trigger.

RDB advantages and disadvantages

advantage:

1. RDB is a compact compressed binary file, on behalf of Redis data snapshot at a certain point in time. Ideal for backup, the whole amount of replication scenarios. For example bgsave backup performed every 6 hours, and copy the file to the remote machine or RDB file system (e.g. HDFS), for disaster recovery.
2. Redis load RDB to recover data much faster than AOF way.

Disadvantages:

1. RDB way no way to do real-time data persistence / second-level persistence. Bgsave have to perform the operation because the fork to create a child process, belongs to the heavyweights operations, frequently performed high cost of each run.
2. RDB file saved using a specific binary format, Redis version of evolution have multiple versions of RDB format, there is not compatible with older versions of the new version of Redis service RDB format problems.

RDB is not suitable for real-time persistent problem, Redis provides a way to solve the AOF persistence.

AOF persistence

AOF (append only file) persistence: independent log recorded each write command, and then re-execute the purpose of AOF file command to achieve data recovery restart.

AOF main role is to solve the real-time data persistence, persistence Redis is now the mainstream way.

Open AOF, the configuration file by modifying redis.conf

appendonly yes ##默认不开启。

AOF file name by appendfilename configuration settings, the default file name appendonly.aof. RDB save path with persistence in a consistent manner, the configuration specified by dir.

FIG AOF works as follows:

1. All the write command will be appended to the aof_buf (buffer).
2. AOF do synchronization buffer corresponding to the hard disk according to the policy.
3. With AOF files increases, the need for regular files for AOF rewrite, to achieve the purpose of compression.
4. When the Redis server restarts, you can load AOF file for data recovery.

Tips

** Scene: ** AOF files may exist at the end it is incomplete, such as a sudden power failure cause the machine AOF end of the file write command incomplete.

Solution:

1. For AOF file format error, first be backed up , and then using redis-check-aof --fix order to repair, using the difference diff-u comparative data after the repair, find lost data, you can manually modify some completions.
2. Redis provides aof-load-truncated configured to be compatible with our case , enabled by default. When loading AOF, when faced with this problem is ignored and continue to start, while the print log the following warning:

# !!! Warning: short read while loading the AOF file !!!
# !!! Truncating the AOF at offset 397856725 !!!
# AOF loaded anyway because aof-load-truncated is enabled

Persistent optimization

Redis persistence function has been the impact of the high incidence of Redis performance. Mainly in the following aspects

1. fork operation

Causes : For instance Redis OPS high traffic of up to 50,000 or more, if the fork-consuming operation in the second level will slow down tens of thousands of Redis command execution, delayed impact on the online application is obvious. Normally each fork Processed GB should consume about 20 milliseconds. You can check the statistics in the last fork latest_fork_usec index obtaining time-consuming operation, the unit microseconds info stats.

Optimization :

1. High priority physical machine or virtualization support fork operation, avoid the use of Xen.
2. Redis instance control the maximum available memory, fork with time-consuming proportional to the amount of memory, recommended that each line Redis instance memory control in less than 10GB.
3. The rational allocation of Linux memory allocation strategy to avoid physical memory failure due to insufficient fork.
4. Reduce the frequency of operation of the fork, as to relax the AOF automatically trigger timing, to avoid unnecessary replication of the full amount.

CPU

• Analysis of CPU overhead. Sub-process is responsible for the data written to the file in the batch process, a process that belongs to the CPU-intensive operation, usually the child of a single-core CPU utilization close to 90%.
• CPU consumption optimization. Redis is a CPU-intensive services, do not bind the single-core CPU operation. Since the sub-processes consuming CPU, and the parent process can produce single-core competition for resources.
• Do not CPU-intensive services and other deployed together, resulting in excessive CPU competition. If you deploy multiple Redis instance, try to ensure that only one child process to rewrite the work.

RAM

Memory consumption optimization:

1. With the same CPU optimization, Redis If you deploy multiple instances, try to ensure that only one child process at work.
2. Avoid doing child process reprogramming operation in a large number of written, this will result in a copy of the parent process to maintain a large number of pages, resulting in memory consumption.

hard disk

Optimization methods are as follows:

• Do not load the hard drive and other high service deployment together. The: storage services, message queuing services.
• AOF will consume a lot of time rewriting the hard disk IO, you can open the configuration no-appendfsync-on-rewrite, turned off by default. It means no fsync operations during the AOF rewrite.
• When opening AOF Redis function for high traffic scenario is written, if the use of ordinary mechanical disk, generally written in throughput 100MB / s or so, when the primary synchronization bottleneck Redis instance AOF on the hard disk.
• For the case where a plurality of single Redis instance, can be configured to different instances of divided files AOF disk storage, disk write sharing pressure.

Note: Configure no-appendfsync-on-rewrite = yes, you may lose data during the entire AOF rewrite In extreme cases, you need to configure whether the data security decisions.

AOF additional obstruction

When you turn AOF persistence, common hard disk synchronization strategy is everysec , for balance of performance and data security. For this way, Redis use another thread of execution per second fsync synchronize hard disk. When the system hard disk resources are busy blocking the main thread will cause Redis,

Blocking flow analysis:

1. The main thread is responsible for writing AOF buffer.
2. AOF thread is responsible for the implementation of a synchronous disk operations per second and record the last synchronization time.
3. The main thread is responsible for AOF than the last time synchronization: If success since the last synchronization time within 2 seconds, return directly to the main thread. If the success from the last synchronization time over two seconds, the main thread will block until the sync is complete.

This article is over! Like a friend little attention and praise, thanks for the support. Continuously updated more dry goods