A must-know series for back-end architects: message queues and asynchronous processing

Author: Zen and the Art of Computer Programming

1 Introduction

Introduction to Message Queue

In the microservice architecture model, due to the large number of applications and the complicated service calling relationships, it eventually leads to the decline of availability, performance and other indicators. In order to solve these problems, a distributed architecture model needs to be adopted. One of the important components is the message queue, which can be used as middleware to implement data communication between services.

Message queue is an advanced technology that allows loosely coupled communication between applications, processes messages asynchronously, and ensures data consistency through reliable delivery. Message queues are divided into point-to-point models and publish/subscribe models. The point-to-point model is similar to a "customer service" in a telephone system that directly delivers information to customers, while the publish/subscribe model allows multiple consumers to receive the same message at the same time.

Why use message queue?

Let's take a look at why we use message queues. There are mainly the following reasons:

1. Cut peaks and fill valleys. When traffic suddenly increases, requests can be concentrated through the message queue and then processed one by one, reducing the pressure on the back-end server and improving the system's responsiveness.

2. Traffic cuts. Under normal circumstances, when the number of website visits reaches a certain level, the database query speed will become a bottleneck. By adding a message queue, a large number of requests can be buffered, thereby reducing the frequency of queries to the database.

3. Data consistency. Message queue can ensure data consistency. For example, after a user order is successfully created, a message can be sent to the corresponding message queue to notify other systems to update inventory.

4. reliability. Message queues can ensure reliable transmission of messages. For example, messages may be lost due to network delays or routing problems, but the message queue retry mechanism can ensure that the message finally arrives.

5. Scalability. Message queues are highly scalable. By adding message queue servers, applications can be scaled out and quickly respond to changing workloads.

6. Asynchronous processing. Message queue supports asynchronous processing, which can improve user experience. For example, when a user uploads a file and notifies the background for processing through the message queue, there is no need to wait for the background processing to complete before returning a response, which can respond to user requests faster.

In short, message queue is a method of information exchange between distributed systems, which can be used to help realize communication and collaboration between services under a microservice architecture. Therefore, it is very necessary for back-end architects to master message queues.

Message queue and asynchronous processing principles and practices

The principle of message queue and asynchronous processing

Synchronous blocking and asynchronous non-blocking

During program execution, if a task takes too long, it will affect the running of other tasks. So, how to optimize program running efficiency? One way is through an asynchronous non-blocking approach. The so-called asynchronous non-blocking means that some tasks that do not involve IO operations do not wait for the results to be returned immediately, but continue to execute the next task; other tasks are postponed until the IO operations are completed before processing. The advantage of this is that it can improve CPU utilization and shorten processing time.

So, how to implement asynchronous non-blocking programming? The simplest way is to use an event loop. Usually, we can start an event loop in a thread and continuously detect whether there are new tasks that need to be processed in the loop. If so, get the task, execute the task, and put the result into the callback function or future object. When the IO operation is completed, the result is obtained through the future object and the corresponding callback function continues to be executed. Although this method is simple, it lacks flexibility and cannot cope with complex scenarios. For example, if one task depends on the results of another task, it cannot be executed sequentially.

Therefore, in order to deal with complex scenarios, a new abstract concept is introduced - coroutine, which is a smaller execution unit than a thread. A coroutine consists of one or more stack frames, which can pause execution and save the current state. When a coroutine needs to perform IO operations, it can switch to other coroutines without affecting the execution of other coroutines. Therefore, coroutines make writing asynchronous code easy and very efficient.

Message Queuing Principles and Functions

As mentioned earlier, message queue is a method of information exchange between distributed systems, which provides asynchronous processing capabilities. In a distributed system, due to the intricate service calling relationships, back-end services need to communicate with each other. Therefore, the message queue can help various services in the distributed system achieve asynchronous communication, including RPC (Remote Procedure Call) calls between services, data flow, etc.

Basically, a message queue is a queue that contains messages to be processed. The producer is the program that adds messages to the queue, and the consumer is the program that takes the messages from the queue and processes them. The message queue has two main functions: First, it ensures the delivery of messages. In practical applications, as long as the producer and consumer are configured correctly, the message queue can ensure the delivery of messages. Second, it provides asynchronous processing. In other words, the producer only produces messages, and the consumer does not wait for the producer to finish sending, and can process the next message by itself.

RabbitMQ message queue

RabbitMQ is currently the most popular open source message broker software, developed by Erlang language. It supports multiple message routing strategies, HA (High Availability) high-availability deployment, and multiple interface protocols such as HTTP and AMQP. RabbitMQ can be well integrated with many language platforms and systems, so it is widely used in enterprise-level environments. Let's take a look at how RabbitMQ works.

The above figure depicts the workflow of RabbitMQ. First, the producer sends a message to Exchange, which decides which Queue to deliver the message to. Exchange delivers the message to the corresponding Queue according to its own type (direct, topic, fanout), routing key (for direct, topic type exchange) or headers (for headers type exchange). Each Queue has a consumer listening, which reads and processes messages in a FIFO (First In First Out) manner. When the consumer finishes processing a message, RabbitMQ automatically removes the message from the queue so that the producer can send new messages again.

Message queue usage scenarios

There are many application scenarios for message queues. Here we list a few common scenarios:

1. Task queue. Message queues can be used to handle slow or idempotent tasks. For example, submit a task to the image processing task queue instead of waiting for the task to complete synchronously.

2. Executed asynchronously. Many web frameworks provide mechanisms for asynchronous execution. For example, many web frameworks provide asynchronous request interfaces, and user requests can be executed asynchronously through message queues without occupying server resources.

3. Loosely coupled architecture. In the microservice architecture, communication and collaboration between services are a very important part. Message queues can provide a loosely coupled architecture. For example, in an e-commerce website, user behavior data can be stored in a message queue, and the order system can consume this data asynchronously.

4. Distributed transactions. Under the microservice architecture, in order to ensure data consistency, the transaction support provided by message queue can be used. For example, user A purchases product X and generates an order, but due to network problems there is a backlog of messages. At this time, user B also purchases product Using message queue transactions can ensure that all participating parties' data operations succeed or fail.

Message queue and asynchronous processing practice

RabbitMQ installation and configuration

The installation of RabbitMQ depends on the Erlang language, so you need to install Erlang first. Download address http://www.erlang.org/downloads. After the download is complete, just follow the installation documentation to install it. After the installation is completed, open the command line and enter the `erl` command. A welcome message appears to indicate a successful installation.

Then, we can install RabbitMQ. RabbitMQ official website https://www.rabbitmq.com/download.html, download the Windows version compressed package and extract it to the specified directory.

Before configuring RabbitMQ, you need to create a user, password and virtual host. Open the bin directory of RabbitMQ and double-click to open it rabbitmqctl.bat. Enter the following command to create user admin:

rabbitmqctl add_user admin password # 创建用户名和密码
rabbitmqctl set_user_tags admin administrator # 设置权限

We can then create a virtual host vhost:

rabbitmqctl add_vhost my_vhost # 创建虚拟主机
rabbitmqctl set_permissions -p my_vhost admin ".*" ".*" ".*" # 设置权限

Next, we need to modify the configuration file C:\Program Files\RabbitMQ Server\rabbitmq.confto specify our username, password, and virtual host:

[
  {rabbit, [
    {tcp_listen_port,     5672},
    {disk_free_limit,      50000000}, %% 设置磁盘限制，超过这个值，RabbitMQ会自动删除部分消息
    {delegate_count,       10},    %% 设置最大连接数，默认500
    {default_user,         "guest"},
    {default_pass,         "guest"},
    {default_vhost,        "/"     },
    {loopback_users,       []}
  ]},

  {rabbitmq_management, [
    {listener, [{port,          15672},
                {ip,            "127.0.0.1"}]}]}
].

[{sasl, [
      {sasl_error_logger, {file,"<path to log directory>"}},
      {errlog_type, error},
      {error_logger_mf_dir, "<path to log directory>"}, % Logs dir
      {error_logger_mf_maxbytes, 10485760}, % Max file size of logs
      {error_logger_mf_maxfiles, 5} % Max number of files in logs
    ]}].

%% 用户名、密码、虚拟主机配置
{rabbit,[
        {default_user, << "admin" >>},
        {default_pass, << "password" >>},
        {default_vhost, <<"my_vhost">>}
       ]}.

Finally, we restart RabbitMQ:

net stop RabbitMQ && net start RabbitMQ

Using Python to operate RabbitMQ

First, we need to install the pika module.

pip install pika

We first create two Python scripts, one producer and one consumer. The producer simulates sending messages to the queue, and the consumer reads messages from the queue and prints them out.

The producer's code is as follows:

import time
import random
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

queue_name = 'hello'

for i in range(10):
    message = str(i)+str(random.randint(0,10))+'hello world!'
    channel.basic_publish(exchange='', routing_key=queue_name, body=message)
    print(" [x] Sent {}".format(message))
    time.sleep(1)

connection.close()

The consumer code is as follows:

import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

queue_name = 'hello'

def callback(ch, method, properties, body):
    print(" [x] Received {}".format(body))

channel.basic_consume(queue=queue_name, on_message_callback=callback, auto_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()

connection.close()

Running these two scripts, you can see that the producer sends 10 messages to the queue, and the consumer reads them and prints them.