From principle to actual combat, teach you how to use RabbitMQ in your project

Hello everyone, I am Lou Tsai.

The RabbitMQ article was written before, but the example given at that time was the Demo version. This article mainly integrates RabbitMQ into the technical project based on the theoretical knowledge written before.

Not BB, on the article directory:

Next, let's review the theoretical knowledge first. If you are already clear about this knowledge, you can skip directly to the actual combat part.

1. Message queue

1.1 Message queue mode

There are currently two main modes of message queues, namely "point-to-point mode" and "publish/subscribe mode".

peer-to-peer mode

A specific message can only be consumed by one consumer, and multiple producers can send messages to the same message queue, but when a message is processed by a messager, the message will be locked or removed from the queue And other consumers cannot process the message.

It should be noted that if a consumer fails to process a message, the message system will generally put the message back into the queue so that other consumers can continue processing.

 If you want to learn automated testing, here I recommend a set of videos for you. This video can be said to be the number one automated testing tutorial on the entire network at station B. At the same time, the number of people online has reached 1,000, and there are notes to collect and share with you. Dashen Technical Exchange: 798478386   

[Updated] The most detailed collection of practical tutorials for automated testing of Python interfaces taught by station B (the latest version of actual combat)_哔哩哔哩_bilibili [Updated] The most detailed collection of practical tutorials for automated testing of Python interfaces taught by station B (actual combat) The latest version) has a total of 200 videos, including: 1. Why should interface automation be done in interface automation, 2. The overall view of request in interface automation, 3. Interface combat in interface automation, etc. UP hosts more exciting videos, please pay attention to UP account . https://www.bilibili.com/video/BV17p4y1B77x/?spm_id_from=333.337.search-card.all.click 

publish/subscribe model

A single message can be fetched and processed concurrently by multiple subscribers. In general, there are two types of subscriptions:

• Ephemeral subscriptions : These subscriptions only exist while the consumer is up and running. Once the consumer exits, the corresponding subscription and unprocessed messages are lost.

• Durable (durable) subscription : This subscription will always exist unless it is actively deleted. After the consumer exits, the messaging system will continue to maintain the subscription, and subsequent messages can continue to be processed.

1.2 RabbitMQ characteristics

• Message routing (support) : RabbitMQ can support different types of message routing through different switches;

• Ordered messages (not supported) : When consuming messages, if the consumption fails, the messages will be put back into the queue and then re-consumed, which will cause the messages to be out of order;

• Message timing (very good) : through the delay queue, you can specify the delay time of the message, the expiration time TTL, etc.;

• Fault-tolerant handling (very nice) : Handle message processing failures with delivery retries and a Dead Letter Exchange (DLX);

• Scaling (general) : Scaling is not very intelligent, because even if it is scaled, there is still only one master queue, and only this one master queue can resist the load, so I understand that RabbitMQ's scaling is very weak (personal understanding).

• Persistence (not very good) : Messages without consumption can support persistence. This is to ensure that the messages can be restored when the machine is down, but the consumed messages will be deleted immediately, because RabbitMQ is not designed for to store historical data.

• Message backtracking (support) : Because messages do not support permanent storage, backtracking is naturally not supported.

• High throughput (medium) : Because all requests are executed in the master queue at the end, its design makes the performance of a single machine not reach the 100,000-level standard.

2. A preliminary study on the principle of RabbitMQ

Released in 2007, RabbitMQ is an open source message queuing system developed in Erlang language and implemented based on the AMQP protocol.

2.1 Basic concepts

When it comes to RabbitMQ, you have to mention the AMQP protocol. The AMQP protocol is a binary protocol with modern features. It is an application layer standard advanced message queuing protocol that provides unified messaging services. It is an open standard for application layer protocols and is designed for message-oriented middleware.

First understand several important concepts in the AMQP protocol:

• Server: Receives the connection from the client and implements the AMQP entity service.

• Connection: connection, network connection between application and Server, TCP connection.

• Channel: Channel, message read and write operations are performed in the channel. A client can establish multiple channels, each channel representing a session task.

• Message: Message, the data transmitted between the application and the server, the message can be very simple or complex. It consists of Properties and Body. Properties is the outer packaging, which can modify the message, such as message priority, delay and other advanced features; Body is the content of the message body.

• Virtual Host: virtual host for logical isolation. There can be several Exchanges and Queues in a virtual host, and there cannot be Exchanges or Queues with the same name in the same virtual host.

• Exchange: The exchange receives messages and routes them to one or more queues according to routing rules. If the route is not available, either return it to the producer, or discard it directly. There are four commonly used types of switches commonly used by RabbitMQ: direct, topic, fanout, and headers, which will be described in detail later.

• Binding: binding, virtual connection between the exchange and the message queue, the binding can contain one or more RoutingKey.

• RoutingKey: Routing key. When the producer sends a message to the switch, it will send a RoutingKey to specify the routing rules, so that the switch knows which queue to send the message to. The routing key is usually a string separated by ".", such as "com.rabbitmq".

• Queue: Message queue, used to store messages for consumption by consumers.

2.2 Working principle

The AMQP protocol model consists of three parts: producer, consumer and server. The execution process is as follows:

1. The producer connects to the server, establishes a connection, and opens a channel.

2. Producers declare exchanges and queues, set related properties, and bind exchanges and queues through routing keys.

3. Consumers also need to perform operations such as establishing a connection and opening a channel to receive messages.

4. The producer sends a message to the virtual host in the server.

5. The switch in the virtual host selects routing rules according to the routing key and sends them to different message queues.

6. Consumers who subscribe to the message queue can get the message and consume it.

2.3 Commonly used switches

There are four types of switches commonly used by RabbitMQ: direct, topic, fanout, and headers:

• Direct Exchange: As you can see in the text, direct exchange means that the exchange needs to be bound to a queue, requiring that the message exactly match a specific routing key. To put it simply, it is one-to-one, point-to-point sending.

• Fanout Exchange: This type of exchange requires binding queues to the exchange. A message sent to an exchange is forwarded to all queues bound to that exchange. Much like subnet broadcasting, every host on the subnet gets a copy of the message. Simply put, it is publish and subscribe.

• Topic Exchange: If translated directly, it is called a topic exchange. If it is translated from the usage above, it may be called a wildcard exchange, which would be more appropriate. This switch uses wildcards to match and routes to the corresponding queue. There are two types of wildcards: "*" and "#". It should be noted that the "." symbol must be added before the wildcard.

  • * symbol: There is and only one word is matched. For example, a.* can match "ab" and "ac", but not "abc".

  • # symbol: Match one or more words. For example, "rabbit.#" can match not only "rabbit.ab", "rabbit.a", but also "rabbit.abc".

• Headers Exchange: This kind of switch is relatively not used so much. It is a little different from the above three types. Its routing does not use routingKey for routing matching, but routing by matching the key value carried in the request header. To create a queue, you need to set the binding header information. There are two modes: full match and partial match. As shown in the figure above, the switch will match the key value bound to the queue according to the key value carried in the header information sent by the producer, and route to the corresponding queue.

3. RabbitMQ environment construction

Because I use a Mac, I can directly refer to the official website:

https://www.rabbitmq.com/install-homebrew.html

It should be noted that it must be executed first:

brew update

Then execute:

brew install rabbitmq

Before brew update was not executed, when brew install rabbitmq was executed directly, various strange errors would be reported, most of which were "403 Forbidde".

But when you execute "brew install rabbitmq", other programs will be installed automatically. If you install Rabbitmq using source code, because starting the service depends on the erlang environment, you need to install erlang manually, but the official has already done it for you with one click. Isn't it great to automatically install all the programs that Rabbitmq depends on!

The output of the final successful execution is as follows:

Start the service:

# 启动方式1：后台启动
brew services start rabbitmq
# 启动方式2：当前窗口启动
cd /usr/local/Cellar/rabbitmq/3.8.19
rabbitmq-server

Enter in the browser:

http://localhost:15672/

The RabbitMQ background management interface will appear (both user name and password are guest):

Installed through brew, one line of command to get it done, really fragrant!

4. RabbitMQ integration

4.1 Preliminary work

Add account:

## 添加账号
./rabbitmqctl add_user admin admin
## 添加访问权限
./rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"
## 设置超级权限
./rabbitmqctl set_user_tags admin administrator

Pom introduces dependencies:

<dependency>
    <groupId>com.rabbitmq</groupId>
    <artifactId>amqp-client</artifactId>
    <version>5.5.1</version>
</dependency>

 

4.2 Code implementation

core code

First complete a ConnectionFactory singleton, each machine has its own ConnectionFactory to prevent initialization every time (in later iterations, I will remove this and make it into a connection pool).

/**
 * @author Louzai
 * @date 2023/5/10
 */
public class RabbitmqUtil {

    /**
     * 每个key都有自己的工厂
     */
    private static Map<String, ConnectionFactory> executors = new ConcurrentHashMap<>();

    /**
     * 初始化一个工厂
     *
     * @param host
     * @param port
     * @param username
     * @param passport
     * @param virtualhost
     * @return
     */
    public static ConnectionFactory init(String host,
                                  Integer port,
                                  String username,
                                  String passport,
                                  String virtualhost) {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setHost(host);
        factory.setPort(port);
        factory.setUsername(username);
        factory.setPassword(passport);
        factory.setVirtualHost(virtualhost);
        return factory;
    }

    /**
     * 工厂单例，每个key都有属于自己的工厂
     *
     * @param key
     * @param host
     * @param port
     * @param username
     * @param passport
     * @param virtualhost
     * @return
     */
    public static ConnectionFactory getOrInitConnectionFactory(String key,
                                                               String host,
                                                               Integer port,
                                                               String username,
                                                               String passport,
                                                               String virtualhost) {
        ConnectionFactory connectionFactory = executors.get(key);
        if (null == connectionFactory) {
            synchronized (RabbitmqUtil.class) {
                connectionFactory = executors.get(key);
                if (null == connectionFactory) {
                    connectionFactory = init(host, port, username, passport, virtualhost);
                    executors.put(key, connectionFactory);
                }
            }
        }
        return connectionFactory;
    }
}

Get RabbitmqClient:

/**
 * @author Louzai
 * @date 2023/5/10
 */
@Component
public class RabbitmqClient {

    @Autowired
    private RabbitmqProperties rabbitmqProperties;

    /**
     * 创建一个工厂
     * @param key
     * @return
     */
    public ConnectionFactory getConnectionFactory(String key) {
        String host = rabbitmqProperties.getHost();
        Integer port = rabbitmqProperties.getPort();
        String userName = rabbitmqProperties.getUsername();
        String password = rabbitmqProperties.getPassport();
        String virtualhost = rabbitmqProperties.getVirtualhost();
        return RabbitmqUtil.getOrInitConnectionFactory(key, host, port, userName,password, virtualhost);
    }
}

focus! Knock on the blackboard! ! ! Here is the core logic of RabbmitMQ.

The type of switch we use is Direct Exchange. This switch needs to be bound to a queue and requires that the message exactly match a specific routing key. Simply put, it is one-to-one, point-to-point sending.

As for why we don’t use the broadcast and topic switch mode, because the usage scenario of the technical school is to send a single message, and the point-to-point sending and consumption mode can fully meet our needs.

The following 3 methods are very simple:

• Send message: get factory -> create link -> create channel -> declare switch -> send message -> close link;

• Consume messages: get factory -> create link -> create channel -> determine message queue -> bind queue to switch -> accept and consume messages;

• Consumption message perpetual mode: Consume RabbitMQ messages in non-blocking mode.

@Component
public class RabbitmqServiceImpl implements RabbitmqService {

    @Autowired
    private RabbitmqClient rabbitmqClient;

    @Autowired
    private NotifyService notifyService;

    @Override
    public void publishMsg(String exchange,
                           BuiltinExchangeType exchangeType,
                           String toutingKey,
                           String message) throws IOException, TimeoutException {
        ConnectionFactory factory = rabbitmqClient.getConnectionFactory(toutingKey);

        // TODO: 这种并发量起不来，需要改造成连接池

        //创建连接
        Connection connection = factory.newConnection();
        //创建消息通道
        Channel channel = connection.createChannel();

        // 声明exchange中的消息为可持久化，不自动删除
        channel.exchangeDeclare(exchange, exchangeType, true, false, null);

        // 发布消息
        channel.basicPublish(exchange, toutingKey, null, message.getBytes());

        System.out.println("Publish msg:" + message);
        channel.close();
        connection.close();
    }

    @Override
    public void consumerMsg(String exchange,
                            String queue,
                            String routingKey) throws IOException, TimeoutException {
        ConnectionFactory factory = rabbitmqClient.getConnectionFactory(routingKey);

        // TODO: 这种并发量起不来，需要改造成连接池

        //创建连接
        Connection connection = factory.newConnection();
        //创建消息信道
        final Channel channel = connection.createChannel();
        //消息队列
        channel.queueDeclare(queue, true, false, false, null);
        //绑定队列到交换机
        channel.queueBind(queue, exchange, routingKey);

        Consumer consumer = new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties,
                                       byte[] body) throws IOException {
                String message = new String(body, "UTF-8");
                System.out.println("Consumer msg:" + message);

                // 获取Rabbitmq消息，并保存到DB
                // 说明：这里仅作为示例，如果有多种类型的消息，可以根据消息判定，简单的用 if...else 处理，复杂的用工厂 + 策略模式
                notifyService.saveArticleNotify(JsonUtil.toObj(message, UserFootDO.class), NotifyTypeEnum.PRAISE);

                channel.basicAck(envelope.getDeliveryTag(), false);
            }
        };
        // 取消自动ack
        channel.basicConsume(queue, false, consumer);
    }

    @Override
    public void processConsumerMsg() {
        System.out.println("Begin to processConsumerMsg.");

        Integer stepTotal = 1;
        Integer step = 0;

        // TODO: 这种方式非常 Low，后续会改造成阻塞 I/O 模式
        while (true) {
            step ++;
            try {
                System.out.println("processConsumerMsg cycle.");
                consumerMsg(CommonConstants.EXCHANGE_NAME_DIRECT, CommonConstants.QUERE_NAME_PRAISE,
                        CommonConstants.QUERE_KEY_PRAISE);
                if (step.equals(stepTotal)) {
                    Thread.sleep(10000);
                    step = 0;
                }
            } catch (Exception e) {

            }
        }
    }
}

Here is just an example. If you want to use it in a production environment, what problems do you think there are?  You think about it first, and I'll tell you at the end of the article.

call entry

In fact, we used Java's built-in asynchronous calling method before. In order to facilitate verification, I migrated the function of article likes to RabbitMQ. As long as it is likes, we will follow the RabbitMQ mode.

// 点赞消息走 RabbitMQ，其它走 Java 内置消息机制
if (notifyType.equals(NotifyTypeEnum.PRAISE) && rabbitmqProperties.getSwitchFlag()) {
    rabbitmqService.publishMsg(
            CommonConstants.EXCHANGE_NAME_DIRECT,
            BuiltinExchangeType.DIRECT,
            CommonConstants.QUERE_KEY_PRAISE,
            JsonUtil.toStr(foot));
} else {
    Optional.ofNullable(notifyType).ifPresent(notify -> SpringUtil.publishEvent(new NotifyMsgEvent<>(this, notify, foot)));
}

Where is the consumption entrance? In fact, when the program starts, we start RabbitMQ for consumption, and then the whole process keeps running in the program.

@Override
public void run(ApplicationArguments args) {
    // 设置类型转换, 主要用于mybatis读取varchar/json类型数据据，并写入到json格式的实体Entity中
    JacksonTypeHandler.setObjectMapper(new ObjectMapper());
    // 应用启动之后执行
    GlobalViewConfig config = SpringUtil.getBean(GlobalViewConfig.class);
    if (webPort != null) {
        config.setHost("http://127.0.0.1:" + webPort);
    }
    // 启动 RabbitMQ 进行消费
    if (rabbitmqProperties.getSwitchFlag()) {
        taskExecutor.execute(() -> rabbitmqService.processConsumerMsg());
    }
    log.info("启动成功，点击进入首页: {}", config.getHost());
}

4.3 Demonstration

We clicked the "Like" button multiple times to trigger the RammitMQ message to be sent.

You can also see the sent and consumed messages through the log.

I rely on! Lots of unclosed links. . .

There are still a bunch of channels that are not closed. . .

It is estimated that after running for a while, all the memory will be used up, and the machine will crash, how can it be broken? The answer is the connection pool!

4.4 Code branches

In order to facilitate everyone to learn the process of function evolution, each module will open a separate branch, including the later upgraded version:

• Code warehouse: https://github.com/itwanger/paicoding

• Code branch: feature/add_rabbitmq_20230506

If you need to run RabbitMQ, the following configuration needs to be changed to true, because the code defaults to false.

5 Epilogue

This article, let everyone know the basic principles of RabbitMQ and how to integrate RabbitMQ, but it cannot be used in the actual production environment, but this is indeed the first version I wrote, and it is purely for fun, because there are There are still many problems.

I will briefly list:

1. You need to add a connection pool to Connection, otherwise the memory will continue to be consumed, and the machine will definitely not be able to handle it;

2. The consumption method of RabbitMQ needs to be transformed, because the method of while + sleep is too simple and rude;

3. If the consumption task hangs up, you need to have a consumption mechanism to restart RabbitMQ;

4. If the machine hangs up, the messages inside RabbitMQ cannot be lost after restarting.

If you are also very interested in the above questions, you can directly based on the branch feature/add_rabbitmq_20230506, and then give me a PR, technology, I like to learn while playing.