13 military rules for performance optimization of Redis

In this article, we will use the following methods to improve the running speed of Redis:

Shorten the storage length of key-value pairs;
Use the lazy free (delayed deletion) feature;
Set the expiration time of the key value;
Disable long time-consuming query commands;
Use slowlog to optimize time-consuming commands;
Use Pipeline to manipulate data in batches;
Avoid simultaneous failure of a large amount of data;
Client use optimization;
Limit Redis memory size;
Use a physical machine instead of a virtual machine to install the Redis service;
Check the data persistence strategy;
Disable THP feature;
Use distributed architecture to increase read and write speed.

1. Shorten the storage length of key-value pairs

The length of key-value pairs is inversely proportional to performance. For example, let’s do a set of performance tests for writing data. The execution results are as follows:

It can be seen from the above data that when the key is unchanged, the larger the value, the slower the operation efficiency, because Redis uses different internal codes for storage of the same data type. For example, there are three internal codes for strings: int (integer encoding), raw (string encoding to optimize memory allocation), embstr (dynamic string encoding), this is because the author of Redis wants to achieve a balance of efficiency and space through different encodings, but the larger the amount of data, the greater the amount of data used The more complicated the internal code is, and the more complex the internal code, the lower the performance of storage.

This is only the speed of writing. When the content of the key-value pair is large, it will bring several other problems:

The larger the content, the longer the persistence time is required, and the longer the suspension time is, the lower the performance of Redis will be;
The larger the content, the more content is transmitted on the network, the longer it takes, and the lower the overall operating speed;
The larger the content, the more memory it occupies, and the more frequent the memory elimination mechanism will be triggered, which brings more operational burden to Redis.

Therefore, while ensuring complete semantics, we should shorten the storage length of key-value pairs as much as possible, and if necessary, serialize and compress the data before storing. Taking Java as an example, we can use protostuff or kryo for serialization. Compression can be Use snappy.

2. Use the lazy free feature

The lazy free feature is a new and very useful feature of Redis 4.0, which can be understood as lazy deletion or delayed deletion. It means to provide the function of asynchronous delayed key value release when deleting, and put the key value release operation in a separate sub-thread of BIO (Background I/O) to reduce the blockage of the Redis main thread by deletion, which can be effective To avoid the performance and availability problems caused by deleting the big key.

lazy free corresponds to 4 scenarios, which are all closed by default:

lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
slave-lazy-flush no

The meanings they represent are as follows:

lazyfree-lazy-eviction: Indicates whether to enable lazy free mechanism to delete when Redis running memory exceeds maxmeory;
lazyfree-lazy-expire: indicates that the key value of the expiration time is set, and whether to open the lazy free mechanism to delete after the expiration;
lazyfree-lazy-server-del: Some instructions will carry an implicit del key operation when processing existing keys, such as the rename command. When the target key already exists, Redis will delete the target key first. If these targets The key is a big key, which will cause blocking deletion. This configuration indicates whether to open the lazy free mechanism for deletion in this scenario;
Slave-lazy-flush: Perform full data synchronization for the slave (slave node). Before loading the master's RDB file, the slave will run fluxhall to clean up its data. It indicates whether the lazy free mechanism is enabled for deletion at this time.

It is recommended to enable the lazyfree-lazy-eviction, lazyfree-lazy-expire, lazyfree-lazy-server-del and other configurations, so as to effectively improve the execution efficiency of the main thread.

3. Set the expiration time of the key value

We should set a reasonable expiration time for the key value according to the actual business situation, so that Redis will help you automatically clear the expired key-value pair to save memory usage and avoid excessive accumulation of key values and frequent memory triggers Elimination strategy.

4. Disable long time-consuming query commands

The time complexity of most of Redis read and write commands is between O(1) and O(N). The official document has a time complexity description for each command, address: https://redis.io/commands, As shown below:

Among them, O(1) means it can be used safely, and O(N) should be careful, and N means uncertain. The larger the data, the slower the query speed may be. Because Redis only uses one thread for data query, if these instructions take a long time, it will block Redis and cause a lot of delay.

To avoid the impact of O(N) commands on Redis, you can start with the transformation from the following aspects:

Decided to prohibit the use of the keys command;
To avoid querying all members at once, use the scan command to traverse in batches and cursors;
Strictly control the data size of Hash, Set, Sorted Set and other structures through mechanisms;
Put sorting, union, intersection and other operations on the client to reduce the operating pressure of the Redis server;
When deleting (del) a large data, it may take a long time, so it is recommended to use the asynchronous delete method to unlink, which will start a new thread to delete the target data without blocking the main Redis thread.

5. Use slowlog to optimize time-consuming commands

We can use the slowlog function to find the most time-consuming Redis commands and perform related optimizations to improve the running speed of Redis. There are two important configuration items for slow queries:

slowlog-log-slower-than : Used to set the evaluation time of slow query, that is to say, commands that exceed this configuration item will be treated as slow operations and recorded in the slow query log. Its execution unit is microseconds (1 second is equal to 1000000 microseconds);
slowlog-max-len : Used to configure the maximum number of records in the slow query log.

We can make the corresponding configuration according to the actual business situation. The slow log is stored in the slow query log in reverse order in the order of insertion. We can use it slowlog get n to get the relevant slow query log, and then find the business corresponding to these slow queries for related optimization.

6. Use Pipeline to manipulate data in bulk

Pipeline (pipeline technology) is a batch processing technology provided by the client to process multiple Redis commands at one time, thereby improving the performance of the entire interaction.

We use Java code to test the performance comparison between Pipeline and normal operations. The test code of Pipeline is as follows:

publicclass PipelineExample {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
// 记录执行开始时间
long beginTime = System.currentTimeMillis();
// 获取 Pipeline 对象
Pipeline pipe = jedis.pipelined();
// 设置多个 Redis 命令
for (int i = 0; i < 100; i++) {
pipe.set("key" + i, "val" + i);
pipe.del("key"+i);
}
// 执行命令
pipe.sync();
// 记录执行结束时间
long endTime = System.currentTimeMillis();
System.out.println("执行耗时：" + (endTime - beginTime) + "毫秒");
}
}

The execution result of the above program is:

Execution time: 297 milliseconds

The common operation code is as follows:

publicclass PipelineExample {
public static void main(String[] args) {
Jedis jedis = new Jedis("127.0.0.1", 6379);
// 记录执行开始时间
long beginTime = System.currentTimeMillis();
for (int i = 0; i < 100; i++) {
jedis.set("key" + i, "val" + i);
jedis.del("key"+i);
}
// 记录执行结束时间
long endTime = System.currentTimeMillis();
System.out.println("执行耗时：" + (endTime - beginTime) + "毫秒");
}
}

The execution result of the above program is:

Execution time: 17276 milliseconds

From the above results, it can be seen that the execution time of the pipeline is 297 milliseconds, while the execution time of the ordinary command is 17276 milliseconds. The pipeline technology is about 58 times faster than the ordinary execution.

7. Avoid simultaneous failure of a large amount of data

Redis expired key value deletion uses a greedy strategy. It will perform 10 expired scans per second. This configuration can be configured in redis.conf. The default value is hz 10that Redis will randomly select 20 values and delete the expired keys in these 20 keys. If the proportion of expired keys exceeds 25%, repeat this process, as shown in the following figure:

If a large number of caches expire at the same time in a large system, Redis will continue to scan and delete expired dictionaries multiple times in a loop, until the expired key values in the expired dictionary are sparsely deleted, and the entire execution process will cause Redis to fail. There is obvious lag in reading and writing. Another reason for the lag is that the memory manager needs to reclaim memory pages frequently, so it will consume a certain amount of CPU.

In order to avoid this kind of lag, we need to prevent a large number of caches from expiring at the same time. The simple solution is to add a specified range of random numbers on the basis of the expiration time.

8. Client use optimization

In the use of the client, in addition to using the Pipeline technology as much as possible, we also need to pay attention to using the Redis connection pool as much as possible, instead of frequently creating and destroying Redis connections, so as to reduce the number of network transmissions and reduce unnecessary call instructions.

9. Limit Redis memory size

In the 64-bit operating system, the memory size of Redis is unlimited, that is, the configuration items maxmemory <bytes> are commented out, which will cause the use of swap space to exchange space when the physical memory is insufficient, and when you worry about the system using Redis When the memory paging is moved to the swap space, it will block the Redis process, causing Redis to delay, thereby affecting the overall performance of Redis. Therefore, we need to limit the memory size of Redis to a fixed value. When Redis reaches this value, the memory elimination strategy will be triggered. There are 8 memory elimination strategies after Redis 4.0 :

noeviction : Do not eliminate any data. When the memory is insufficient, new operations will report an error. Redis default memory elimination strategy;
allkeys-lru : Eliminate the longest unused key value in the entire key value;
allkeys-random : randomly eliminate any key value;
Volatile-lru : Eliminate the longest unused key value among all the key values with expiration time;
Volatile-random : randomly eliminate any key value that has an expiration time;
volatile-ttl : Prioritize the elimination of keys that expire earlier.

Two new elimination strategies have been added in Redis 4.0 version:

volatile-lfu : Eliminate the least used key value among all the keys with expiration time;
allkeys-lfu : Eliminate the least used key value in the entire key value.

Among them, allkeys-xxx means eliminating data from all key values, and volatile-xxx means eliminating data from key values with expired keys.

We can set it according to the actual business situation. The default elimination strategy does not eliminate any data, and an error will be reported when adding it.

10. Use physical machines instead of virtual machines

Run the Redis server in a virtual machine, because it shares a physical network port with a physical machine, and a physical machine may have multiple virtual machines running, so there will be very bad performance in terms of memory usage and network latency. We can ./redis-cli --intrinsic-latency 100 Check the delay time through the command. If you have high requirements on the performance of Redis, you should deploy the Redis server directly on the physical machine as much as possible.

11. Check the data persistence strategy

Redis's persistence strategy is to copy memory data to the hard disk so that disaster recovery or data migration can be performed, but maintaining this persistence function requires a lot of performance overhead.

After Redis 4.0, Redis has 3 ways of persistence:

RDB (Redis DataBase, snapshot mode) writes the memory data at a certain moment into the disk in a binary manner;
AOF (Append Only File, file append mode), records all operation commands, and appends to the file in the form of text;
Hybrid persistence method, a new method added after Redis 4.0. Hybrid persistence combines the advantages of RDB and AOF. When writing, first write the current data in the form of RDB to the beginning of the file, and then add subsequent The operation commands are stored in the file in AOF format, which not only guarantees the speed of Redis restart, but also reduces the risk of data loss.

RDB and AOF persistence have their pros and cons. RDB may cause data loss for a certain period of time, while AOF will affect the startup speed of Redis due to the large file size. In order to have the advantages of both RDB and AOF, Redis 4.0 has added Hybrid persistence method, so when we must perform persistence operations, we should choose the hybrid persistence method.

You can use config get aof-use-rdb-preamble commands to query whether hybrid persistence is enabled , and the execution results are shown in the following figure:

Among them, yes means that hybrid persistence has been turned on, no means that it is turned off, and the default value of Redis 5.0 is yes. If it is another version of Redis, you first need to check whether hybrid persistence has been turned on. If it is turned off, you can turn it on in the following two ways:

Open via command line
Open by modifying the Redis configuration file

① Open via command line

The command config set aof-use-rdb-preamble yes execution result is shown in the figure below:

The disadvantage of the command line setting configuration is that after restarting the Redis service, the setting configuration will become invalid.

② Open by modifying the Redis configuration file

Find the redis.conf file in the root path of Redis, and change the configuration file aof-use-rdb-preamble no to the aof-use-rdb-preamble yes following figure:

After the configuration is complete, you need to restart the Redis server for the configuration to take effect, but the configuration file is modified every time the Redis service is restarted, the configuration information will not be lost.

It should be noted that in the business that does not have to be persisted, persistence can be turned off, which can effectively improve the running speed of Redis without intermittent stuck.

12. Disable THP feature

The Linux kernel adds the Transparent Huge Pages (THP) feature to the 2.6.38 kernel, which supports 2MB allocation of large memory pages, which is enabled by default.

When THP is turned on, the speed of fork will slow down. After fork, each memory page changes from 4KB to 2MB, which will greatly increase the memory consumption of the parent process during rewriting. At the same time, the copy memory page unit caused by each write command is enlarged by 512 times, which will slow down the execution time of the write operation, resulting in slow query of a large number of write operations. For example, simple incr commands will also appear in slow queries, so Redis recommends that this feature be disabled. The method of disabling is as follows:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

To make the THP configuration still take effect after the machine is restarted, you can add it to /etc/rc.local echo never > /sys/kernel/mm/transparent_hugepage/enabled.

13. Use a distributed architecture to increase read and write speed

Redis distributed architecture has three important means:

Master-slave synchronization
Sentinel mode
Redis Cluster

Using the master-slave synchronization function, we can put the writing on the main library for execution, and transfer the reading function to the slave service, so we can process more requests per unit of time, thereby improving the overall running speed of Redis.

The sentinel mode is an upgrade of the master-slave function, but when the master node crashes, the normal use of Redis can be automatically restored without manual intervention.

Redis Cluster is officially launched by Redis 3.0. Redis cluster balances the load pressure of each node by distributing the database to multiple nodes.

Redis Cluster uses virtual hash slot partitions. All keys are mapped to integer slots from 0 to 16383 according to the hash function. The calculation formula is: slot = CRC16(key) & 16383. Each node is responsible for maintaining a part of the slots and the keys mapped by the slots. Value data. In this way, Redis can distribute the read and write pressure from one server to multiple servers, so the performance will be greatly improved.

Of these three functions, we only need to use one. There is no doubt that Redis Cluster should be the preferred implementation solution. It can automatically share the read and write pressure to more servers and has automatic disaster recovery capabilities.