Oh my god, that's how Redis Pipeline works

throw, problem

Recently the project encountered such a technical requirement:

The front end obtains the message data of the server through the long polling mechanism (http long polling). The server needs to subscribe to the business messages of all business parties, and then notify the front end.

Long polling, in fact, simply means that the front end initiates an http request, and the server hangs the current request until it times out or there is something to return before returning. Apollo Configuration Center uses this mechanism to implement configuration update notifications.

But there is such a situation, if the server consumes the message, but the connection between the front end and the server is just disconnected, then the message cannot be notified to the front end.

Therefore, we need to save the messages consumed by the server to ensure that every time the front-end initiates long polling, the message data can be obtained.

If the front-end can directly subscribe to the message queue of the business side, then it has nothing to do with the server. Of course, this is not allowed, the front end cannot connect to our message middleware, and the message data of the business side also needs to be cleaned and processed before it can be sent to the front end.

thinking, plan

Apollo's implementation mechanism is that all configurations are written to the database. Every time a request comes, it will go to the database to check whether there is any data change.

Considering our actual business scenario, our business messages are actually time-sensitive, that is to say, if the message expires, it is useless.

To consider light weight, the first thing that comes to my mind is Redis Lists, which can be used to implement the characteristics of queues, so I will finally use Redis as the storage model for messages.

optimize, code

The Lists data structure of Redis is a simple linked list of strings, sorted by insertion order. Elements can be added at the head (Left) or at the end (Right).

Using this feature, we can implement a queue (first-in, first-out) data structure with the command rpush + lpop or lpush + rpop .

But whether it is lpop or rpop , the list will only pop up one piece of data, which cannot meet our needs to get more than one at a time.

I searched for relevant solutions on the Internet, and then combined with the command documentation on the official website.

A more feasible solution is drawn: lrange + ltrim + pipeline

lrange and ltrim are instructions for Redis Lists

lrange mylist 0 5

Indicates that the element is taken from the head of the queue (Left), with a subscript of 0 and an element with a subscript of 5. If it is a negative number, it means that the tail of the queue (Right) starts to fetch.

ltrim mylist 0 5

The meaning of ltrim is to keep only the elements of the specified range. The above means that only the elements with subscripts 0 - 5 of the list are kept, and the others will be deleted.

Then if you want to delete the first 6 pieces of data, you may need to write:

ltrim mylist 5 -1

A negative number means that the tail of the queue (Right) starts to fetch.

According to the implementation, lrange + ltrim can achieve the effect of obtaining multiple elements from List, why do you need to use pipeline?

In fact, there is an obvious concurrency problem here. These two commands are  not atomic operations for redis  . If other threads delete part of the data in the queue between the threads executing these two commands, then the second When the command is executed, the data in the list is actually wrong.

The pipeline function of Redis can solve the problem we mentioned above.

Most people may be unfamiliar with pipelines, because they are rarely used in business.

Let's take a look at the definition of Redis Pipelining

A Request/Response server can be implemented so that it is able to process new requests even if the client hasn't already read the old responses. This way it is possible to send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step.

This is called pipelining, and is a technique widely in use for many decades. For instance many POP3 protocol implementations already support this feature, dramatically speeding up the process of downloading new emails from the server.

In short, it can support the client to send multiple commands at a time. After the server receives these commands, they will be processed in order and in an atomic transaction.

Obviously, the most obvious advantage of Pipelining is to improve the interaction response time between client and server. Putting several commands in the same request will definitely improve the efficiency compared to each command as a request.

PS: This method is similar to the idea of ​​why I want to obtain multiple list data. When the front-end long polling obtains data results, it needs to return multiple data.

Without further ado, paste the sample code directly:

@Autowired
RedisTemplate<String, Object> redisTemplate;

public void run() throws Exception {
    System.out.println("start......");
    String key = "key:list";
    Integer number = 5;
    // 每次获取5个
    List<Object> objects = redisTemplate.executePipelined(new RedisCallback<Object>() {

        @Override
        public Object doInRedis(RedisConnection connection) throws DataAccessException {
            connection.lRange(key.getBytes(StandardCharsets.UTF_8), 0, number - 1);
            connection.lTrim(key.getBytes(StandardCharsets.UTF_8), number, -1);
            return null;
        }
    });
    System.out.println(Arrays.toString(objects.toArray()));
    System.out.println("end......");
}

redisTemplate provides two types of methods:

• executePipelined(RedisCallback action)
• executePipelined(SessionCallback session)

The functions implemented by the two are similar, except that SessionCallback is more friendly than the package of RedisCallck. The latter is more low-level and the usage is closer to the native command.