Before implementing task queues, let's first understand the benefits of using task queues:
1. Loose coupling. Producers and consumers do not need to know each other's implementation details, but only need to agree on the description format of the task. This allows producers and consumers to be written by different teams using different programming languages.
2. Easy to expand. Consumers can be multiple, and can be distributed in different servers, which can easily reduce the load of a single server.
To implement a queue, it is natural to think of the Redis list type, as well as the LPUSH and RPOP commands. If you want to implement a task queue, you only need to let the producer add tasks to a key using the LPUSH command, and let the consumer continue to use the RPOP command to remove tasks from the key. The pseudocode implementation of Redis is as follows:
# Infinite loop reads the contents of the task queue
loop
$task = RPOP queue
if $task
# If there is a task in the task queue, execute it
execute($task)
else
# If there is no task, wait for 1 second to avoid requesting data too frequently
wait 1 second
So far, a simple task queue implemented using Redis has been written, but there is still one problem: when there are no tasks in the task queue, the consumer will call the RPOP command every second to check whether there are new tasks.
Optimization: With the help of the BRPOP command, consumers can be notified as soon as a new task is added to the queue
The BRPOP command accepts two parameters, the first is the key name, and the second is the timeout in seconds. If no new element is obtained after this time, it will return nil. If the timeout is set to "0", it means that there is no limit to the waiting time, and if no new elements are added to the list, it will block forever.
BRPOP and RPOP commands are similar, the only difference is: when there are no elements in the task list, the BRPOP command will block the connection until a new element is added. The above pseudocode can be optimized to:
loop
# If there are no new tasks in the task queue, the BRPOP command will always block and execute() will not be executed
$task = BRPOP queue, 0
# The return value is an array, the second element of the array is the task we need
execute($task[1])
Queues sometimes need priority. For example, when the system needs to send a confirmation email and a notification email at the same time, the confirmation email should be executed first. The specific scenario is as follows. There are 100,000 users who subscribe to a celebrity's blog. When the blog publishes a new article, the blog will add 100,000 tasks to the task queue to send notification emails. If each email takes 10ms, it will take 100 000 tasks to complete all of them: 100 000 * 10 / 1000 = 1000 seconds (nearly 20 minutes). During the joining period, a new user wants to subscribe to the blog. When he submits his email and sees a webpage prompt to check the confirmation email, the user does not know that the task of the confirmation email sent to him is added to the one that already has 100,000 tasks. In the queue, need to wait nearly 20 minutes for this.
After analyzing the specific scenario, the task of notifying subscribers after publishing a new article is not very urgent, and a delay of 20 minutes is completely acceptable. Therefore, the following conclusion can be drawn: when two tasks of sending confirmation emails and sending notification emails exist at the same time, the former should be executed first. To achieve this, we need to implement a priority queue.
The BRPOP command can accept multiple keys at the same time. The complete command format is: BRPOP key [key ...] timeout, such as: BRPOP queue1 queue2 0. This means that multiple keys are detected at the same time. If one of the keys has an element, The element is popped from that key; if multiple keys have elements, the first element in the first key is taken from left to right.
With this feature, prioritized task queues can be implemented. We use the two keys queue:confirmation.email and queue.notification.email to store the two tasks of sending confirmation emails and sending notification emails, and then modify the pseudocode of the consumer to:
loop
$task =
BRPOP queue:confirmationl.email,
queue:notification.email,
0
execute($task[1])
At this time, once the task of sending the confirmation email is added to the queue.confirmation.email queue, no matter how many tasks there are in queue:notification.email, the consumer will give priority to the notification task of sending the confirmation email.