RabbitMQ: Powerful messaging middleware for reliable messaging

 Messaging middlewares play a key role in modern distributed systems, they provide a reliable and efficient method for asynchronous communication and decoupling. In this blog, we will focus on RabbitMQ, a widely used open source messaging middleware. We'll dive into RabbitMQ's features, how it works, and how you can use it in your applications for reliable messaging.

1. Introduction to RabbitMQ

RabbitMQ is an open source messaging middleware based on AMQP (Advanced Message Queuing Protocol). It provides a reliable, flexible, and extensible messaging mechanism that is widely used in various industries. The core idea of ​​RabbitMQ is that the producer sends the message to the switch, the switch passes the message to the queue according to the routing rules, and then the consumer gets and processes the message from the queue.

2. Related concepts

RabbitMQ is an open source message middleware written in Erlang and implements the Advanced Message Queuing Protocol (AMQP). As a reliable, flexible and scalable messaging system, RabbitMQ provides a reliable mechanism for transferring data between applications.

The following is a detailed explanation of some key features and concepts of RabbitMQ:

1. Message Queues: RabbitMQ uses message queues to store and deliver messages. A message queue is a first-in-first-out (FIFO) data structure that temporarily stores messages in it until a consumer is ready to receive and process them.

2. Producer and Consumer: The sender of the message is called the producer, and the receiver of the message is called the consumer. Producers send messages to queues, and consumers get messages from queues and process them.

3. Queue: Queue is the core part of RabbitMQ, which is the storage and delivery carrier of messages. Producers send messages to the queue, and consumers get messages from the queue. Queues can be persistent, which means that even if the RabbitMQ server is down, messages will not be lost.

4. Exchange (Exchange): The exchange is the router of the message, which sends the message from the producer to the queue. It routes messages to one or more queues according to specific rules. RabbitMQ provides different types of switches, including direct switches, topic switches, broadcast switches, etc.

5. Binding: Binding associates exchanges and queues to define routing rules for messages between exchanges and queues. Bindings specify how messages should be routed from exchanges to queues.

6. Routing Key: A routing key is an attribute that a producer specifies along with a message when it sends it. The exchange uses the routing key to determine which queue to route the message to.

7. Persistence: When queues or messages are marked as persistent, they are persisted to disk to prevent data loss after RabbitMQ restarts.

8. Publish-subscribe mode: RabbitMQ supports publish-subscribe mode, in which a producer sends a message to the switch, and the switch broadcasts the message to all queues bound to it, and then the consumers of each queue can receive and process the message.

9. ACK mechanism: Consumers can use the ACK mechanism to inform RabbitMQ that the message has been successfully received and processed. RabbitMQ will remove the message from the queue only after the consumer explicitly acknowledges it.

10. Message acknowledgment and persistence: RabbitMQ allows producers to request acknowledgment when sending messages. If a message cannot be successfully routed to a queue, or if an error occurs during routing, RabbitMQ will notify the producer. Additionally, message and queue persistence ensures no data loss even after RabbitMQ restarts.

In short, RabbitMQ is a reliable and flexible message middleware. It takes message queue as its core and uses concepts such as switches, queues, and bindings to route and deliver messages. It provides a highly reliable messaging mechanism and rich features, and is widely used in distributed systems, microservice architectures, task queues, and other scenarios to help applications achieve decoupling, asynchronous communication, and reliability assurance.

3. How RabbitMQ works

The working principle of RabbitMQ can be simply summarized as the following steps:

1. The producer sends a message to the exchange:

Producers connect to RabbitMQ and send messages to predefined exchanges. Messages can contain any structured data such as JSON, XML, etc.

String message = "Hello, RabbitMQ!";
channel.basicPublish(exchangeName, routingKey, null, message.getBytes());

2. The switch routes the message to the queue:

An exchange routes messages to one or more queues bound to it according to predefined routing rules. Routing rules can be based on exact match, pattern match, etc.

channel.queueBind(queueName, exchangeName, routingKey);

3. The consumer gets the message from the queue:

Consumers connect to RabbitMQ and subscribe to queues of interest. Once a message arrives in the queue, RabbitMQ will immediately push the message to subscribed consumers for processing.

channel.basicConsume(queueName, true, new DefaultConsumer(channel) {
    @Override
    public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
        String message = new String(body, "UTF-8");
        System.out.println("Received message: " + message);
    }
});

4. The consumer processes the message and sends a confirmation:

After consumers get the message, they can perform corresponding business logic processing. Once the message is successfully processed, the consumer will send an acknowledgment to RabbitMQ that the message has been consumed and RabbitMQ can safely delete the message.

A producer sends messages to an exchange named "my_exchange" and routes messages to a queue named "my_queue" with routing key "my_routing_key". The consumer subscribes to the "my_queue" queue and calls the callback function to process the message when it receives it.

4. Features of RabbitMQ

RabbitMQ provides many powerful features that make it a widely used messaging middleware:

• Persistence : RabbitMQ can persist messages and queues to disk, and messages will not be lost even after the server restarts.
• Flexible routing rules : By using different switch types and routing keys, precise message routing strategies can be achieved.
• Reliability and recoverability : RabbitMQ provides a variety of mechanisms to ensure reliable delivery of messages, such as message acknowledgment, transaction, publisher confirmation, etc.
• Scalability : RabbitMQ supports distributed deployment and cluster mode, which can achieve high throughput and high availability.
• Multi-language client : RabbitMQ provides a variety of officially supported client libraries, such as Java, Python, Ruby, etc., to facilitate developers to use in different language environments.

5. Application Scenarios of RabbitMQ in Practical Applications

Reference article:

【RabbitMQ】What is RabbitMQ? What is RabbitMQ useful for? What are the application scenarios? _Luyao Yezi's Blog-CSDN Blog

RabbitMQ has a wide range of applications in various scenarios, including but not limited to:

• Asynchronous task processing : Send long-time-consuming tasks to RabbitMQ and execute them asynchronously by consumers to improve the response performance and scalability of the system.
• Event-driven architecture : By using RabbitMQ to implement event publishing and subscription, different components can be decoupled by subscribing to events of interest.
• Application decoupling and flow control : By introducing message middleware, different applications can be decoupled, and mechanisms such as flow control and service degradation can be implemented.
• Log collection and analysis : Use RabbitMQ to send log messages in the distributed system to the central log server for centralized management and analysis.

 5.1 Decoupling between services

User orders, inventory processing. [Decoupling between services]

Before using MQ:

When the system is normal, the user places an order, and the order system calls the inventory system to perform a deletion operation. If the operation is successful, it will return a message to remind the order to be placed successfully. When the system is abnormal, the inventory system will be inaccessible, resulting in the inability to execute the order deletion operation, and ultimately lead to the failure of placing the order.

After using MQ:

The order system and the inventory system no longer interact with each other and run independently, achieving the purpose of application decoupling. The order system only needs to write the order message into MQ, and then it can directly execute the next step. At this time, even if the inventory system is abnormal, it will not affect the operation of the order system, and the inventory deletion record of the order will be permanently saved in MQ until the inventory system returns to normal, and subscribe to the order message from MQ for consumption until successful.

Before using MQ:

After using MQ:

5.2  Realize asynchronous communication

User registration, sending mobile phone text messages, emails . [Realize asynchronous communication]

Before using MQ:

The entire operation process is all completed in the main thread. Click User Registration--"Add User in Storage--"Send Email--"Send SMS. Each step needs to wait for the previous step to complete before it can execute. Moreover, the response time of each operation step is not fixed. If there are too many requests, it will take a long time to request the main thread, slow the response, and even cause a crash, which seriously affects the user experience.

After using MQ:

The main thread only needs to process the less time-consuming warehousing operations, and then write the messages to be processed into the MQ message queue, and then subscribe to the messages in the message queue for consumption by different independent mail systems and SMS systems at the same time. In this way, the message queue is used as an intermediary to store and deliver messages, which is not only time-consuming and consumes very few resources, but also a single server can bear more concurrent requests.

 

5.3  Flow peak clipping

Commodity flash sales and snap-ups. 【Flow peak clipping】

Traffic peak shaving is a common scenario in message queues and is generally used in flash sales or group buying activities.

Before using MQ: For flash sales and panic buying activities, the traffic generated by user access will be very large, and even tens of millions of requests will appear in the same period of time. This instantaneous traffic surge, our application system configuration is unbearable , will cause the system to crash directly.

For example: System A usually requests 100 requests per second, and the system runs stably; however, there is a spike activity at 8:00 p.m., and the number of requests increases to 10,000 per second, and the system can process a maximum of 1,000 requests per second, causing the system to crash. 

After using MQ: When a large number of users make spike requests, we reject that huge traffic request at the upper layer of system business processing and transfer it to MQ instead of directly pouring into our interface. Here the MQ message queue acts as a cache.

For example: 1 million users request 5,000 requests per second during the peak period. Writing these 5,000 requests into the MQ system can only process 2,000 requests per second, because MySQL can only handle 2,000 requests; the system pulls 2,000 requests per second. More than you can handle.

Before using MQ:

 After using MQ:

5.4 Other application scenarios:

1. Order processing system:

In an e-commerce platform, RabbitMQ can be used to process orders. When a user places an order, the order information is published to the RabbitMQ switch, and then related consumers get the order message from the queue and process it, such as verifying the order, inventory management, payment, etc.

2. Log collection and distribution:

Suppose you have multiple applications that generate logs and want to process and store them centrally. Each application can publish log messages to an exchange called "log_exchange", and then different consumers subscribe to the exchange and write the log messages to the database or send to the log analysis system.

3. Real-time data transmission:

If there is a real-time monitoring system, the sensor data needs to be transmitted to the monitoring platform in real time for processing and visual display. The sensor publishes the data to the RabbitMQ switch, and the consumer of the monitoring platform subscribes to the switch and processes the data, so as to realize real-time monitoring and alarm functions.

4. Asynchronous task processing:

Suppose there is an application that needs to handle a lot of time-consuming tasks, such as image processing, PDF conversion, etc. The application publishes tasks to the RabbitMQ queue, and then multiple worker nodes act as consumers to obtain tasks from the queue and process them to achieve parallel processing of tasks and reduce the pressure on the main application.

5. Message notification system:

 Suppose in a subscription system, users can subscribe to different topics or events. When a new message is published, RabbitMQ will route the message to the corresponding queue, and then users who subscribe to the queue will receive corresponding notifications. This method can be used to implement email subscription, news push and other functions.

6. Microservice architecture:

In a microservice architecture, messaging and collaboration may be required between different services. Using RabbitMQ can realize decoupling and asynchronous communication between services, and each service can send and receive messages through switches and queues, thus achieving loose coupling between microservices.

6. RabbitMQ installation

Official website address:

RabbitMQ: easy to use, flexible messaging and streaming — RabbitMQ

6.1 brew installation

brew update #更新一下homebrew
brew install rabbitmq   #安装rabbitMQ 

Installation result:

==> Caveats
==> rabbitmq
Management Plugin enabled by default at http://localhost:15672

To restart rabbitmq after an upgrade:
  brew services restart rabbitmq
Or, if you don't want/need a background service you can just run:
  CONF_ENV_FILE="/opt/homebrew/etc/rabbitmq/rabbitmq-env.conf" /opt/homebrew/opt/rabbitmq/sbin/rabbitmq-server

 The installation path of rabbitmq:

/opt/homebrew/opt/rabbitmq

6.2, configure environment variables

1、

vi ~/.bash_profile

2、 

export RABBIT_HOME=${PATH}:/opt/homebrew/opt/rabbitmq
export PATH=${PATH}:$RABBIT_HOME/sbin

3、 

source ~/.bash_profile

6.3, start RabbitMQ

1. Run
rabbitmq-server in the foreground

2. Run
rabbitmq-server-detached in the background

3. View the running status
rabbitmqctl status

4. Start the web plugin
rabbitmq-plugins enable rabbitmq_management

5. Restart
rabbitmq-server restart

5. Close
rabbitmqctl stop

6.4. Access MQ

1. Browser address

http://localhost:15672/
The default username and password are guest

Add user
rabbitmqctl add_user miaojiang 123

Set the user as an administrator
rabbitmqctl set_user_tags miaojiang administrator

Configure users to log in remotely to
rabbitmqctl set_permissions -p "/" miaojaing ".*" ".*" ".*"

View newly added accounts

rabbitmqctl list_users

View the permissions used
rabbitmqctl list_permissions -p /

Seven, Spring Boot project application RabbitMQ

7.1. Add Maven dependency:

Add the RabbitMQ client library dependency to your project's pom.xml file

<!--AMQP依赖，包含RabbitMQ-->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

7.2. Configure RabbitMQ connection:

Add RabbitMQ connection information in the Spring Boot configuration file (application.properties or application.yml).

application.properties:

spring.rabbitmq.host=localhost
spring.rabbitmq.port=5672
spring.rabbitmq.username=guest
spring.rabbitmq.password=guest

Configure mq parameters in application.yml:

spring:
  rabbitmq:
    #设置RabbitMQ的IP地址
    host: localhost
    #设置rabbitmq服务器用户名
    username: guest
    #设置rabbitmq服务器密码
    password: guest
    #设置rabbitmq服务器连接端口
    port: 5672

7.3 Create a switch

custom switch name

Create an exchange called "myExchange"

package com.example.usermanagement.mq;

import org.springframework.amqp.core.*;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class RabbitMQConfig {
    /*
    使用 @Configuration 注解创建一个配置类，并通过 @Bean 注解创建了一个名为 declareExchange 的方法，用于声明创建交换机。请根据实际情况修改交换机名称、类型和持久化设置。
     */

    public static final String EXCHANGE_NAME = "myExchange";

    @Bean
    public Exchange declareExchange() {
        return ExchangeBuilder.directExchange(EXCHANGE_NAME)
                .durable(true)
                .build();
    }
}

7.4 Create a message sender

Create a message sender: Create a message sender class for sending messages to RabbitMQ

package com.example.usermanagement.mq;

import org.springframework.amqp.core.AmqpTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
public class MessageSender{

    private final AmqpTemplate amqpTemplate;
    private final String exchangeName = "myExchange"; // 自定义交换机名称

    @Autowired
    public MessageSender(AmqpTemplate amqpTemplate) {
        this.amqpTemplate = amqpTemplate;
    }

    public void sendMessage(Object message) {
        amqpTemplate.convertAndSend(exchangeName, "", message); // 发送消息到默认交换机和空路由键
    }
}

Notice:

 The sendMessage type uses the Object type

7.5 RabbitMQ management background to add columns

step:

1. Open a browser and enter the URL of the RabbitMQ management background. By default, the URL is http://localhost:15672/. Please make sure your RabbitMQ server is running and on the correct port number.

2. Enter the username and password to log in to the RabbitMQ management background. By default, the username is guest, and the password is also guest. If you have changed your username and password, please log in with your custom credentials.

3. After successfully logging in, you will see the main interface of the RabbitMQ management background. In the top navigation bar, choose QueuesTabs.

4. On Queuesthe page, you will see a list of existing queues. If you want to create a new queue, click Add a new queuethe button.

5. On the page to add a queue, fill in the following information:

   • Name: The name of the queue. Provide a unique name for the queue. (such as myQueue)
   • Durability: Persistence of the queue. Select Yes or No to specify whether the queue should persist across RabbitMQ service restarts.
   • Auto delete: Automatic deletion of the queue. Select Yes or No to specify whether to delete the queue when the last consumer disconnects.
   • Arguments: Additional parameters for the queue. This is optional, you can set some specific parameters for the queue.

6. After filling out the queue information, click Add queuethe button to create the queue.

7. After the creation is successful, you will Queuessee the newly added queue on the page. You can view the details of the queue on this page, including the number of messages, number of consumers, etc.

http://localhost:15672/#/queues

Just add the queue name 

7.6 Calling the producer

1. Injection MessageSenderinstance

@Autowired
private MessageSender messageSender;

2. Call the method where the message needs to be sent messageSender.sendMessage. According to your business logic, you can call this method at an appropriate location. For example, after the order is successfully created, you can add the following code:

messageSender.sendMessage("订单已创建：" + order.getOrderId());

7.7 Create a message receiver

Create a message receiver: Create a message receiver class for processing received RabbitMQ messages.

Here we directly write the logic for processing RabbitMQ messages.

package com.example.usermanagement.mq;

import org.springframework.amqp.rabbit.annotation.Exchange;
import org.springframework.amqp.rabbit.annotation.Queue;
import org.springframework.amqp.rabbit.annotation.QueueBinding;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

@Component
public class MessageReceiver {

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue("your_queue_name"),
            exchange = @Exchange(value = RabbitMQConfig.EXCHANGE_NAME)
//            key = "your_routing_key"
    ))
    public void receiveMessage(Object message) {
        System.out.println("Received message: " + message);
        // 处理消息逻辑
    }
}

Notice:

 The sendMessage type uses the Object type

your_queue_nameReplace with the name of the queue you want to listen to, (such as myQueue)

Replace your_routing_keywith the appropriate routing key (if used)

Eight, MQ interface introduction

Overview: Provides an overview report including server and cluster information, node status, queue and connection summaries, and recent related log entries.
Connections: Displays the client applications currently connected to the RabbitMQ server, including the connection name, protocol, virtual host, and other information.
Channels: Shows active channels on each connection, and some metrics related to each channel, such as number of consumers, number of unacknowledged messages, etc.
Exchanges: Lists all exchanges, including name, type, bound queues, and bound quantity.
Queues (queue): Displays all queues, including information such as name, number of messages, number of consumers, etc. You can also perform some operations through the queue, such as create, delete, empty, etc.
Admin (administrator): Provides some advanced management functions, such as user and authority management, virtual host management, plug-in management, etc.

8.1 Overview

Overview: Provides an overview report including server and cluster information, node status, queue and connection summaries, and recent related log entries.

​

8.2 Connections

Connections: Displays the client applications currently connected to the RabbitMQ server, including the connection name, protocol, virtual host, and other information.

8.3 Channels

Channels: Shows active channels on each connection, and some metrics related to each channel, such as number of consumers, number of unacknowledged messages, etc.

8.3.1 prefetch (prefetch)

In Message Queue (Message Queue), prefetch (prefetch) is an important concept, which is used to control the speed at which consumers get messages from the message queue.

Prefetch refers to getting a certain number of messages into the local cache before the consumer gets the message from the message queue for fast processing by the consumer. In this way, the throughput and efficiency of the system can be improved, and the overhead of network transmission can be reduced.

Specifically, prefetch can help avoid the following situations:

1. Fair dispatching : When multiple consumers process messages in parallel, if there is no prefetch limit, one consumer may get too many messages at one time, resulting in fewer opportunities for other consumers. By setting prefetch, you can ensure that each consumer can only get a certain number of messages to achieve a fairer message distribution.

2. Consumer load balancing : When there are a large number of pending messages in the message queue, consumers may backlog messages due to slow processing speed. By setting an appropriate prefetch value, you can limit the number of messages that consumers get each time, so that messages can be evenly distributed to multiple consumers, thereby achieving load balancing.

In RabbitMQ, the settings of prefetch can basic.qosbe configured through the method. For example, the following code will set the prefetch count to 10:

channel.basic_qos(prefetch_count=10)

Please note that the settings of prefetch should be tuned according to specific application scenarios and system load conditions. Properly setting prefetch can improve the performance and stability of the system.

8.4 Exchanges (switches)

Exchanges: Lists all exchanges, including name, type, bound queues, and bound quantity.

8.5 Queues (queue)

Queues (queue): Displays all queues, including information such as name, number of messages, number of consumers, etc. You can also perform some operations through the queue, such as create, delete, empty, etc.

8.6 Admin (administrator)

Admin (administrator): Provides some advanced management functions, such as user and authority management, virtual host management, plug-in management, etc.