Regarding message queues, we are accustomed to using Rabbitmq and Kafka as message queue middleware. However, when there is only one group of consumers in the consumption queue, it does not need very high reliability, and it is very cumbersome to use middleware. At this time, we can use the characteristics of redis to implement a simple message queue.
asynchronous message queue
The list data structure of Redis is often used as an asynchronous message queue. Use rpush and lpush to operate such as a queue, and use lpop and rpop to dequeue. It
supports multiple producers and multiple consumers to send in and out messages concurrently. Each consumer gets The messages are all distinct list elements, as shown:
> rpush notify-queue apple banba pear
> lpop notify-queue
“apple”
> lpop notify-queue
"banana"
> lpop notify-queue
"pear"
> lpop notify-queue
(nil)
The above is the combination of rpush and lpop, you can also use the combination of lpush and rpop, the effect is the same.
Handling when the queue is empty
The client obtains the message through the pop operation of the queue, and then processes it. After processing, get the message and process it. Such a cycle is the life cycle of the client as a queue consumer.
But if the queue is empty, the client will fall into an infinite loop of pops and keep popping. This kind of empty polling will not only increase the CPU consumption of the client, but also increase the QPS of Redis. If such an empty polling client If there are dozens of nodes, the slow query of Redis may increase significantly.
Usually we use sleep to solve this problem. If the queue is empty, let the thread sleep for a while, which not only reduces the CPU consumption of the client, but also reduces the QPS of Redis.
blocking read
The above sleep method can solve the problem. But there is another small problem, that is, sleep will cause the delay of the message to increase. If there is only 1 consumer, then this delay is 1s. If there are multiple consumers, this delay will be reduced, because the sleep time of consumers is diverged.
Is there any way to solve both the consumption problem and the delay problem? Of course there are, that is blpop/brpop.
The prefix character b of these two instructions represents blocking, that is, blocking reading.
When the blocking read has no data for the column, it will immediately go to sleep, and once the data arrives, it will wake up immediately. Messages have almost zero latency.
Idle connection is automatically disconnected
Although the above solution solves the problem of empty columns, it is not perfect, and it still has the problem of idle connections.
If the thread is blocked all the time, the Redis client connection will become an idle connection. If it is idle for too long, the server will usually actively disconnect the connection to reduce the occupation of idle resources. At this time blpop/brpop will throw an exception. So be careful when writing client-side consumers, and try again if an exception is caught.
Implementation of Delayed Queue
Delayed queues can be implemented through Redis's zset. We serialize the message into a string as the value of zset, and the expiration processing time of the message as the score, and then use multiple threads to poll zset to obtain the expired tasks for processing. Multiple threads are used to ensure availability. If one thread is hung up, there are other threads that can continue processing. Because there are multiple threads, it is necessary to consider concurrent contention tasks to ensure that tasks will not be executed multiple times.
The zrem method of Redis is the key to multi-thread and multi-process competition for tasks. Its return value determines whether the current instance has grabbed the task, because the loop method may be called by multiple threads and processes, and the same task may be called Multiple processes and multiple threads grab it.
At the same time, we need to capture handle_msg to prevent individual characters from handling problems and causing the loop to fail and exit abnormally.
public class RedisDelayingQueue<T> {
static class TaskItem<T> {
public String id;
public T msg;
}
private Type TaskType = new TypeReference<TaskItem<T>>(){
}.getType() ;
private Jedis jedis;
private String queueKey;
public RedisDelayingQueue(Jedis jedis,String queueKey) {
this.jedis = jedis;
this.queueKey = queueKey;
}
public void deplay(T msg) {
TaskItem<T> task = new TaskItem<>();
task.id = UUID.randomUUID().toString();
task.msg = msg;
String s = JSON.toJSONString(task);
jedis.zadd(queueKey,System.currentTimeMillis() + 5000, s);
}
public void loop() {
while (!Thread.interrupted()) {
Set<String> values = jedis.zrangeByScore(queueKey,0,System.currentTimeMillis(),0 , 1);
if (values.isEmpty()) {
try {
Thread.sleep(50);
}catch (InterruptedException e) {
break;
}
continue;
}
// 取出
String s = values.iterator().next();
if (jedis.zrem(queueKey,s) > 0) {
TaskItem<T> task = JSON.parseObject(s,TaskType);
this.handleMsg(task.msg);
}
}
}
public void handleMsg(T msg) {
System.out.println(msg);
}
public static void main(String[] args) {
Jedis jedis = new Jedis("124.221.52.57",6379);
jedis.auth("li12345...");
jedis.select(2);
final RedisDelayingQueue<String> queue = new RedisDelayingQueue<>(jedis,"q-demo");
Thread producer = new Thread() {
public void run() {
for (int i = 0; i < 10; i++) {
queue.deplay("codehole"+i);
}
}
};
Thread consumer = new Thread() {
public void run() {
queue.loop();
}
};
producer.start();
consumer.start();
try {
// producer.join();
Thread.sleep(6000);
consumer.interrupt();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}