0. Preface

Redis Pipeline is a mechanism for batch execution of Redis commands. Normally, when a client sends a command to Redis, it needs to wait for the Redis server to finish executing the command and return the result before sending the next command. However, using pipelines, you can send multiple commands at once on the client side, and then wait for the Redis server to return the results of all commands at once, thus reducing the overhead of multiple network round-trips.

In the pipeline, each command sent by the client is cached by the Redis server instead of being executed immediately. When the client calls a method to execute commands (such as EXEC), the Redis server will execute these commands in the order in which they are sent, and return the execution results to the client at once. This achieves the effect of sending multiple commands at one time and receiving multiple results at one time.

By using pipelines, Redis performance and throughput can be greatly improved. Because pipes allow clients to send multiple commands at once, network latency and connection overhead are reduced. In addition, pipelines also support atomic transaction operations, and multiple commands can be encapsulated in one transaction for execution.

Because pipes are unordered, the order in which the results are returned may not match the order in which the commands were sent. When using pipelines, the client needs to parse and process the results according to the actual situation.

In what scenarios is the redis pipeline feature used?

Redis pipeline features can come into play in the following scenarios:

Batch operations: When a large number of Redis operations need to be performed, using pipelines can combine multiple operations into one network round trip, reducing network latency and connection overhead, and greatly improving performance. For example, batch writing, batch reading, batch deletion, etc.
High throughput: The pipeline can execute multiple Redis operations in parallel, improving the system's concurrent processing capabilities to achieve high throughput data processing.
Data pipeline: When multiple Redis commands need to be executed in sequence, and the execution of subsequent commands depends on the results of previous commands, pipelines can be used to implement pipeline processing of data. For example, operations such as calculation, filtering, and sorting can be completed at once through pipelines.
Batch query: When you need to query the values corresponding to multiple keys, you can use pipelines to merge multiple queries into one query, which reduces the cost of multiple network round-trips and improves query efficiency.
Transaction operation: In Redis, transactions are implemented through MULTI/EXEC commands. Multiple commands can be sent to the Redis server together using pipes to achieve atomic transaction operations. This can reduce the number of network round trips and improve transaction execution efficiency.

The Redis pipeline feature is suitable for scenarios that require high performance, high concurrency, and batch operations, and can effectively improve the operating efficiency of Redis and the overall performance of the system. However, pipeline operations are unordered, and the order of returned results may not be consistent with the order in which commands are sent. Therefore, when using pipelines, the results need to be parsed and processed according to the actual situation.

1.Principle

Redis pipelines can be used with TCP connection multiplexing. When using pipes, you have the option of sending multiple commands on the same TCP connection without having to establish and close the connection each time. This can reduce connection overhead and improve data transmission efficiency.

To explain the performance improvement principle of Redis pipeline from the perspective of TCP connection, it mainly involves the following aspects:

Establishing a TCP connection: When the client establishes a connection with the Redis server, a reliable connection will be established through the TCP three-way handshake. Communication between the client and the server occurs through this TCP connection.
Message transmission: In the Redis pipeline, multiple command requests sent by the client will be cached in the request queue instead of being sent to the Redis server immediately. These requests are cached locally on the client.
Batch sending: When the client calls a method to execute a command (such as EXEC), the client will send the commands in the request queue to the Redis server at once. This reduces the number of network round-trips and improves performance. The client sends commands batch by batch to the server through the TCP connection.
Execute commands: After the Redis server receives the commands sent by the client, it will execute these commands in sequence in the order of the commands in the request queue. After all commands are executed, the execution results are returned to the client at once.
Close the connection: When all commands in the pipeline have been executed, the client can choose to close the TCP connection or continue to send other commands. If the connection is not closed, the client can continue to use the TCP connection for communication in subsequent operations.

The principle of Redis pipeline is based on the establishment of TCP connection and message transmission mechanism. By caching multiple command requests locally on the client and then sending them to the Redis server at once, the cost of network round-trips is reduced and performance is improved. At the same time, by maintaining the TCP connection, you can continue to use the pipeline for subsequent operations without closing the connection.

1.1 Processing mechanism of redis pipeline characteristics

The pipeline implementation principle of Redis mainly relies on the following two mechanisms:

Request queue: Command requests sent by the client will first enter a request queue instead of being sent to the Redis server immediately. The request queue caches requests in the order in which the commands are sent, keeping the order of the commands unchanged.
Batch execution: When the client calls a method to execute commands (such as EXEC), the Redis server will execute these commands in sequence according to the order of the commands in the request queue, and return the execution results to the client at once. This achieves the effect of sending multiple commands at one time and receiving multiple results at one time.

The specific implementation process is as follows:

The client sends commands: The client sends multiple commands to the Redis server. These commands will be added to the request queue.
REDIS_PIPELINE command: The client sends a special REDIS_PIPELINE command to the Redis server to start using the pipeline.
Request queue cache command: After the Redis server receives the REDIS_PIPELINE instruction, it adds the command sent by the client to the request queue.
Execute the command and return the result: When the client calls the method of executing the command (such as EXEC), the Redis server will execute the commands in sequence in the request queue. After all commands are executed, the execution results are returned to the client at once.

By using the request queue and batch execution mechanism, Redis's pipeline achieves the effect of sending multiple commands at one time and receiving multiple results at one time, thereby reducing the overhead of multiple network round-trips and improving performance and throughput. At the same time, the pipeline also supports atomic transaction operations, and multiple commands can be encapsulated in one transaction for execution.

Using redis pipeline optimization example

It is common to find a large number of examples of using loops to operate redis in projects. Using a loop to update the cache individually has a big performance problem when there is a large amount of data.

Give a simple example

Suppose there is a requirement to update the user's cache information in batches based on the user ID list. The original implementation was to use a loop to update each user's cached information individually, as shown below:

import redis.clients.jedis.Jedis;

public class CacheUpdater {
    
    
    private Jedis jedis;

    public CacheUpdater() {
    
    
        jedis = new Jedis("localhost", 6379);
    }

    public void updateCache(List<String> userIds) {
    
    
        for (String userId : userIds) {
    
    
            User user = getUserFromDB(userId); // 从数据库获取用户信息
            updateCache(user); // 更新缓存信息
        }
    }

    private void updateCache(User user) {
    
    
        String cacheKey = "user:" + user.getId();
        jedis.set(cacheKey, user.toJson());
    }

    private User getUserFromDB(String userId) {
    
    
        // 从数据库中获取用户信息的逻辑
    }
}

In the above code, each cycle will update the cache information of a user separately, resulting in network communication each time and poor performance.

The code optimized using the Redis pipeline is as follows:

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;

public class CacheUpdater {
    
    
    private Jedis jedis;

    public CacheUpdater() {
    
    
        jedis = new Jedis("localhost", 6379);
    }

    public void updateCache(List<String> userIds) {
    
    
        Pipeline pipeline = jedis.pipelined();
        for (String userId : userIds) {
    
    
            User user = getUserFromDB(userId); // 从数据库获取用户信息
            updateCache(user, pipeline); // 更新缓存信息
        }
        pipeline.syncAndReturnAll();
    }

    private void updateCache(User user, Pipeline pipeline) {
    
    
        String cacheKey = "user:" + user.getId();
        pipeline.set(cacheKey, user.toJson());
    }

    private User getUserFromDB(String userId) {
    
    
        // 从数据库中获取用户信息的逻辑
    }
}

In the optimized code, a Redis pipeline Pipelineobject is created pipeline, and then the pipeline setcommands are used in the loop to batch update the cache information. Finally, pipeline.syncAndReturnAll()execute all pipeline commands at once.

By using Redis pipeline optimization, network communication overhead can be reduced and performance improved.

3. Springboot uses redis pipeline example

Using the Redis pipeline in Spring Boot can be achieved with the help of RedisTemplate and SessionCallback.

How to use the Redis pipeline to query the latest vehicle location GPS information in batches.
Inject RedisTemplate and use executePipelined method in getLatestGpsInfo method to perform Redis pipeline operations. In the pipeline, traverse the vehicle ID list and use the get method of the RedisConnection object to obtain the GPS information of the corresponding vehicle.

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;

import java.util.ArrayList;
import java.util.List;

@Service
public class VehicleService {
    
    
    
    @Autowired
    private RedisTemplate<String, Object> redisTemplate;

    public List<GpsInfo> getLatestGpsInfo(List<String> vehicleIds) {
    
    
        List<GpsInfo> gpsInfoList = redisTemplate.executePipelined(new RedisCallback<List<GpsInfo>>() {
    
    
            @Override
            public List<GpsInfo> doInRedis(RedisConnection connection) {
    
    
                for (String vehicleId : vehicleIds) {
    
    
                    connection.get(redisTemplate.getStringSerializer().serialize("gps:" + vehicleId));
                }
                return null;
            }
        });

        // 解析返回结果
        List<GpsInfo> result = new ArrayList<>();
        for (Object gpsInfoObj : gpsInfoList) {
    
    
            if (gpsInfoObj != null) {
    
    
                String gpsStr = redisTemplate.getStringSerializer().deserialize((byte[]) gpsInfoObj);
                GpsInfo gpsInfo = parseGpsInfo(gpsStr);
                result.add(gpsInfo);
            }
        }

        return result;
    }

    private GpsInfo parseGpsInfo(String gpsStr) {
    
    
        // 解析GPS信息的逻辑
        // ...
        return gpsInfo;
    }
}

4. References

Redis official documentation https://redis.io/topics/pipelining
Redis protocol https://redis.io/topics/protocol
The advantages and uses of Redis pipeline https://blog.csdn.net/u012439416/article/details/80291006
The implementation principle of Redis pipeline https://juejin.cn/post/6844903955324334093

5. Source code address

https://github.com/wangshuai67/icepip-springboot-action-examples
https://github.com/wangshuai67/Redis-Tutorial-2023

6. Redis series of articles from entry to mastery

[Practice] Detailed explanation of the use of redis pipeline pipeline

Article directory