Dark Horse Microservice Advanced Chapter MQ Advanced Features

The last day of the advanced chapter of Dark Horse Microservices course.

Table of contents

RabbitMQ advanced features

1. Message reliability

1.1. Producer message confirmation

1.1.1. Modify configuration

1.1.2. Define Return callback

1.1.3. Define ConfirmCallback

1.2. Message persistence

1.2.1.Switch persistence

1.2.2. Queue persistence

1.2.3. Message persistence

1.2.3. Message persistence

1.3.Consumer message confirmation

1.3.1. Demonstrate none mode

1.3.2. Demonstrate auto mode

1.4. Retry mechanism for consumption failure

1.4.1. Local retry

1.4.2. Failure strategy

1.5. Summary

2. Dead letter switch

2.1. First introduction to dead letter switches

2.1.1.What is a dead letter switch?

2.1.2. Use the dead letter switch to receive dead letters (extension)

2.1.3. Summary

2.2.TTL

2.2.1. Dead letter switch receiving timeout dead letter

2.2.2. Declare a queue and specify TTL

2.2.3. When sending a message, set TTL

2.2.4. Summary

2.3. Delay queue

2.3.1.Install the DelayExchange plug-in

2.3.2.DelayExchange principle

2.3.3. Using DelayExchange

1) Declare the DelayExchange switch

2) Send message

2.3.4. Summary

3. Lazy queue

3.1. Message accumulation problem

3.2. Lazy queue

3.2.1. Set lazy-queue based on the command line

3.2.2. Declaring lazy-queue based on @Bean

3.2.3. Declare LazyQueue based on @RabbitListener

3.3. Summary

4.MQ cluster

4.1.Cluster classification

4.2. Ordinary cluster

4.2.1. Cluster structure and characteristics

4.2.2.Deployment

4.3. Mirror cluster

4.3.1. Cluster structure and characteristics

4.3.2.Deployment

4.4. Arbitration queue

4.4.1. Cluster characteristics

4.4.2.Deployment

4.4.3.Java code creates arbitration queue

4.4.4.SpringAMQP connects to MQ cluster


RabbitMQ advanced features

During the use of message queues, we face many practical problems that need to be considered:

1. Message reliability

From sending a message to being received by the consumer, multiple processes are handled:

 

Each of these steps may result in message loss. Common reasons for loss include:

  • Lost while sending:

    • The message sent by the producer is not delivered to the exchange

    • The message does not reach the queue after reaching the exchange

  • MQ is down and the queue loses messages

  • After receiving the message, the consumer crashes without consuming it.

In response to these problems, RabbitMQ provides solutions:

  • Producer confirmation mechanism

  • mq persistence

  • Consumer confirmation mechanism

  • Failure retry mechanism

Resource link:

https://pan.baidu.com/s/1fVFV20UBbd07YnkHbALyWw?pwd=5p49#list/path=%2F

Below we will demonstrate each step through a case.

First, import the demo project provided by the data:

 The project structure is as follows:

1.1. Producer message confirmation

RabbitMQ provides a publisher confirm mechanism to avoid message loss during sending to MQ. This mechanism must assign a unique ID to each message. After the message is sent to MQ, a result will be returned to the sender, indicating whether the message was successfully processed.

There are two ways to return results:

  • publisher-confirm, sender confirmation

    • The message is successfully delivered to the switch and ack is returned.

    • The message is not delivered to the switch and nack is returned.

  • publisher-return, sender receipt

    • The message is delivered to the switch, but is not routed to the queue. Returns ACK and the reason for routing failure.

Notice:

 

1.1.1. Modify configuration

First, modify the application.yml file in the publisher service and add the following content:

spring:
  rabbitmq:
    publisher-confirm-type: correlated
    publisher-returns: true
    template:
      mandatory: true

 

illustrate:

  • publish-confirm-type: Turn on publisher-confirm. Two types are supported here:

    • simple: Wait for the confirm result synchronously until timeout

    • correlated: Asynchronous callback, define ConfirmCallback, MQ will call back this ConfirmCallback when returning the result

  • publish-returns: Turn on the publish-return function, which is also based on the callback mechanism, but defines ReturnCallback

  • template.mandatory: Define the strategy when message routing fails. true, call ReturnCallback; false: discard the message directly

1.1.2. Define Return callback

Each RabbitTemplate can only be configured with one ReturnCallback, so it needs to be configured when the project is loaded:

Modify the publisher service and add one:

package cn.itcast.mq.config;

import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.annotation.Configuration;

@Slf4j
@Configuration
public class CommonConfig implements ApplicationContextAware {
    @Override
    public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
        // 获取RabbitTemplate
        RabbitTemplate rabbitTemplate = applicationContext.getBean(RabbitTemplate.class);
        // 设置ReturnCallback
        rabbitTemplate.setReturnCallback((message, replyCode, replyText, exchange, routingKey) -> {
            // 投递失败,记录日志
            log.info("消息发送失败,应答码{},原因{},交换机{},路由键{},消息{}",
                     replyCode, replyText, exchange, routingKey, message.toString());
            // 如果有业务需要,可以重发消息
        });
    }
}

1.1.3. Define ConfirmCallback

ConfirmCallback can be specified when sending a message, because the logic of each business processing confirm success or failure is not necessarily the same.

In the cn.itcast.mq.spring.SpringAmqpTest class of the publisher service, define a unit test method:

public void testSendMessage2SimpleQueue() throws InterruptedException {
    // 1.消息体
    String message = "hello, spring amqp!";
    // 2.全局唯一的消息ID,需要封装到CorrelationData中
    CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());
    // 3.添加callback
    correlationData.getFuture().addCallback(
        result -> {
            if(result.isAck()){
                // 3.1.ack,消息成功
                log.debug("消息发送成功, ID:{}", correlationData.getId());
            }else{
                // 3.2.nack,消息失败
                log.error("消息发送失败, ID:{}, 原因{}",correlationData.getId(), result.getReason());
            }
        },
        ex -> log.error("消息发送异常, ID:{}, 原因{}",correlationData.getId(),ex.getMessage())
    );
    // 4.发送消息
    rabbitTemplate.convertAndSend("task.direct", "task", message, correlationData);

    // 休眠一会儿,等待ack回执
    Thread.sleep(2000);
}

1.2. Message persistence

Producer confirmation can ensure that the message is delivered to the RabbitMQ queue, but after the message is sent to RabbitMQ, if the machine goes down suddenly, the message may also be lost.

To ensure that messages are safely saved in RabbitMQ, the message persistence mechanism must be enabled.

  • Switch persistence

  • Queue persistence

  • Message persistence

1.2.1.Switch persistence

The switch in RabbitMQ is non-persistent by default and will be lost after mq restarts.

In SpringAMQP, switch persistence can be specified through code:

@Bean
public DirectExchange simpleExchange(){
    // 三个参数:交换机名称、是否持久化、当没有queue与其绑定时是否自动删除
    return new DirectExchange("simple.direct", true, false);
}

In fact, switches declared by SpringAMQP are persistent by default.

You can see in the RabbitMQ console that persistent switches will be marked withD:

1.2.2. Queue persistence

Queues in RabbitMQ are non-persistent by default and will be lost after mq is restarted.

In SpringAMQP, switch persistence can be specified through code:

@Bean
public Queue simpleQueue(){
    // 使用QueueBuilder构建队列,durable就是持久化的
    return QueueBuilder.durable("simple.queue").build();
}

In fact, queues declared by SpringAMQP are persistent by default.

You can see in the RabbitMQ console that the persistent queues will be marked withD:

1.2.3. Message persistence

When sending a message using SpringAMQP, you can set the message properties (MessageProperties) and specify delivery-mode:

  • 1: Non-persistent

  • 2: Sustainability

Specify with java code:

1.2.3. Message persistence

When sending a message using SpringAMQP, you can set the message properties (MessageProperties) and specify delivery-mode:

  • 1: Non-persistent

  • 2: Sustainability

Specify with java code:

 By default, any message sent by SpringAMQP is persistent and does not need to be specifically specified.

1.3.Consumer message confirmation

RabbitMQ is aburn after reading mechanism. RabbitMQ will delete the message immediately after it confirms that it has been consumed by the consumer.

RabbitMQ uses consumer receipts to confirm whether the consumer has successfully processed the message: after the consumer obtains the message, it should send an ACK receipt to RabbitMQ to indicate that it has processed the message.

Imagine this scenario:

  • 1) RabbitMQ delivers messages to consumers

  • 2) After the consumer obtains the message, it returns ACK to RabbitMQ

  • 3) RabbitMQ deletes messages

  • 4) The consumer is down and the message has not been processed yet

In this way, the message is lost. Therefore, the timing of the consumer returning ACK is very important.

SpringAMQP allows three confirmation modes to be configured:

•Manual: Manual ack, you need to call the API to send ack after the business code is completed.

•auto: automatic ack. Spring monitors the listener code for exceptions. If there is no exception, ack is returned; if an exception is thrown, nack is returned.

•none: Turn off ack. MQ assumes that the consumer will successfully process the message after getting it, so the message will be deleted immediately after delivery.

It can be seen from this:

  • In none mode, message delivery is unreliable and may be lost.

  • The auto mode is similar to the transaction mechanism. When an exception occurs, nack is returned and the message is rolled back to mq; if there is no exception, ack is returned.

  • Manual: You can judge when to ack based on the business situation.

Generally, we just use the default auto.

1.3.1. Demonstrate none mode

Modify the application.yml file of the consumer service and add the following content:

spring:
  rabbitmq:
    listener:
      simple:
        acknowledge-mode: none # 关闭ack

Modify the method in the SpringRabbitListener class of the consumer service to simulate a message processing exception:

@RabbitListener(queues = "simple.queue")
public void listenSimpleQueue(String msg) {
    log.info("消费者接收到simple.queue的消息:【{}】", msg);
    // 模拟异常
    System.out.println(1 / 0);
    log.debug("消息处理完成!");
}

The test can find that when the message processing throws an exception, the message is still deleted by RabbitMQ.

1.3.2. Demonstrate auto mode

Change the confirmation mechanism to auto again:

spring:
  rabbitmq:
    listener:
      simple:
        acknowledge-mode: auto # 关闭ack

Break the point at the abnormal position and send the message again. When the program is stuck at the breakpoint, you can find that the message status is unack (undetermined status):

After the exception is thrown, because Spring will automatically return nack, the message returns to the Ready state and is not deleted by RabbitMQ:

 

1.4. Retry mechanism for consumption failure

When a consumer encounters an exception, the message will continue to be requeue (requeue) to the queue, and then resent to the consumer. Then it will be abnormal again, requeue again, and the infinite loop will cause MQ's message processing to soar and bring unnecessary pressure:

How to do it?

1.4.1. Local retry

We can use Spring's retry mechanism to use local retry when the consumer encounters an exception, instead of unlimited requeue to the mq queue.

Modify the application.yml file of the consumer service and add the following content:

spring:
  rabbitmq:
    listener:
      simple:
        retry:
          enabled: true # 开启消费者失败重试
          initial-interval: 1000 # 初识的失败等待时长为1秒
          multiplier: 1 # 失败的等待时长倍数,下次等待时长 = multiplier * last-interval
          max-attempts: 3 # 最大重试次数
          stateless: true # true无状态;false有状态。如果业务中包含事务,这里改为false

 

Restart the consumer service and repeat the previous test. It can be found:

  • After retrying 3 times, SpringAMQP will throw an exception AmqpRejectAndDontRequeueException, indicating that the local retry is triggered.

  • Check the RabbitMQ console and find that the message has been deleted, indicating that SpringAMQP finally returned ack and mq deleted the message.

in conclusion:

  • When local retry is enabled, an exception is thrown during message processing and will not be requeued to the queue, but will be retried locally on the consumer.

  • After the maximum number of retries is reached, Spring will return ack and the message will be discarded.

1.4.2. Failure strategy

In the previous test, the message was discarded after reaching the maximum number of retries, which was determined by Spring's internal mechanism.

After the retry mode is enabled and the number of retries is exhausted, if the message still fails, the MessageRecovery interface needs to be used to handle it, which contains three different implementations:

  • RejectAndDontRequeueRecoverer: After the retries are exhausted, directly reject and discard the message. This is the default

  • ImmediateRequeueMessageRecoverer: After the retries are exhausted, nack is returned and the message is re-enqueued.

  • RepublishMessageRecoverer: After the retries are exhausted, the failed message is delivered to the specified switch.

A more elegant solution is RepublishMessageRecoverer. After failure, the message will be delivered to a designated queue dedicated to storing abnormal messages, and subsequent manual processing will be centralized.

1) Define the switch and queue for processing failed messages in the consumer service

@Bean
public DirectExchange errorMessageExchange(){
    return new DirectExchange("error.direct");
}
@Bean
public Queue errorQueue(){
    return new Queue("error.queue", true);
}
@Bean
public Binding errorBinding(Queue errorQueue, DirectExchange errorMessageExchange){
    return BindingBuilder.bind(errorQueue).to(errorMessageExchange).with("error");
}

 

2) Define a RepublishMessageRecoverer and associate the queue and switch

@Bean
public MessageRecoverer republishMessageRecoverer(RabbitTemplate rabbitTemplate){
    return new RepublishMessageRecoverer(rabbitTemplate, "error.direct", "error");
}

Complete code:

package cn.itcast.mq.config;

import org.springframework.amqp.core.Binding;
import org.springframework.amqp.core.BindingBuilder;
import org.springframework.amqp.core.DirectExchange;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.rabbit.retry.MessageRecoverer;
import org.springframework.amqp.rabbit.retry.RepublishMessageRecoverer;
import org.springframework.context.annotation.Bean;

@Configuration
public class ErrorMessageConfig {
    @Bean
    public DirectExchange errorMessageExchange(){
        return new DirectExchange("error.direct");
    }
    @Bean
    public Queue errorQueue(){
        return new Queue("error.queue", true);
    }
    @Bean
    public Binding errorBinding(Queue errorQueue, DirectExchange errorMessageExchange){
        return BindingBuilder.bind(errorQueue).to(errorMessageExchange).with("error");
    }

    @Bean
    public MessageRecoverer republishMessageRecoverer(RabbitTemplate rabbitTemplate){
        return new RepublishMessageRecoverer(rabbitTemplate, "error.direct", "error");
    }
}

1.5. Summary

How to ensure the reliability of RabbitMQ messages?

  • Enable the producer confirmation mechanism to ensure that the producer's messages can reach the queue

  • Turn on the persistence function to ensure that messages will not be lost in the queue before being consumed.

  • Turn on the consumer confirmation mechanism to auto, and spring will complete the ack after confirming that the message processing is successful.

  • Enable the consumer failure retry mechanism and set up MessageRecoverer. After multiple retries fail, the message will be delivered to the abnormal switch for manual processing.

2. Dead letter switch

2.1. First introduction to dead letter switches

2.1.1.What is a dead letter switch?

What is a dead letter?

When a message in a queue meets one of the following conditions, it can become a dead letter:

  • The consumer uses basic.reject or basic.nack to declare consumption failure, and the requeue parameter of the message is set to false

  • The message is an expired message and no one will consume it after timeout.

  • The queue message to be delivered is full and cannot be delivered.

If the queue containing dead letters is configured with the dead-letter-exchange attribute and specifies a switch, then the dead letters in the queue will be delivered to this switch, and this switch is called < /span> (Dead Letter Exchange, check DLX). Dead Letter Exchange

As shown in the figure, a message was rejected by the consumer and became a dead letter:

Because simple.queue is bound to the dead letter switch dl.direct, the dead letter will be delivered to this switch:

 

If this dead letter exchange is also bound to a queue, the message will eventually enter the queue that stores dead letters:

 

In addition, when the queue delivers the dead letter to the dead letter switch, it must know two pieces of information:

  • Dead letter switch name

  • The RoutingKey bound to the dead letter switch and the dead letter queue

This ensures that the delivered message can reach the dead-letter switch and be correctly routed to the dead-letter queue.

2.1.2. Use the dead letter switch to receive dead letters (extension)

In the failure retry strategy, the default RejectAndDontRequeueRecoverer will send reject to RabbitMQ after the number of local retries is exhausted, and the message becomes a dead letter and is discarded.

We can add a dead letter switch to simple.queue and bind a queue to the dead letter switch. In this way, the message will not be discarded after it becomes dead letter, but will eventually be delivered to the dead letter switch and routed to the queue bound to the dead letter switch.

 

We define a set of dead letter switches and dead letter queues in the consumer service:  

// 声明普通的 simple.queue队列,并且为其指定死信交换机:dl.direct
@Bean
public Queue simpleQueue2(){
    return QueueBuilder.durable("simple.queue") // 指定队列名称,并持久化
        .deadLetterExchange("dl.direct") // 指定死信交换机
        .build();
}
// 声明死信交换机 dl.direct
@Bean
public DirectExchange dlExchange(){
    return new DirectExchange("dl.direct", true, false);
}
// 声明存储死信的队列 dl.queue
@Bean
public Queue dlQueue(){
    return new Queue("dl.queue", true);
}
// 将死信队列 与 死信交换机绑定
@Bean
public Binding dlBinding(){
    return BindingBuilder.bind(dlQueue()).to(dlExchange()).with("simple");
}

2.1.3. Summary

What kind of news becomes a dead letter?

  • The message is rejected by the consumer or returns nack

  • Message timed out and was not consumed

  • The queue is full

What are the usage scenarios of dead letter switches?

  • If the queue is bound to a dead letter switch, the dead letter will be delivered to the dead letter switch;

  • You can use the dead letter exchange to collect all messages that consumers failed to process (dead letters) and hand them over to manual processing to further improve the reliability of the message queue.

2.2.TTL

If a message in a queue times out and is not consumed, it will become a dead letter. There are two situations of timeout:

  • The queue where the message is located has a timeout set

  • The message itself has a timeout set

2.2.1. Dead letter switch receiving timeout dead letter

In the SpringRabbitListener of the consumer service, define a new consumer and declare the dead letter switch and dead letter queue:

@RabbitListener(bindings = @QueueBinding(
    value = @Queue(name = "dl.ttl.queue", durable = "true"),
    exchange = @Exchange(name = "dl.ttl.direct"),
    key = "ttl"
))
public void listenDlQueue(String msg){
    log.info("接收到 dl.ttl.queue的延迟消息:{}", msg);
}

2.2.2. Declare a queue and specify TTL

To set a timeout for a queue, you need to configure the x-message-ttl attribute when declaring the queue:

@Bean
public Queue ttlQueue(){
    return QueueBuilder.durable("ttl.queue") // 指定队列名称,并持久化
        .ttl(10000) // 设置队列的超时时间,10秒
        .deadLetterExchange("dl.ttl.direct") // 指定死信交换机
        .build();
}

Note that this queue has the dead letter switch set todl.ttl.direct

Declare the switch and bind ttl to the switch:

@Bean
public DirectExchange ttlExchange(){
    return new DirectExchange("ttl.direct");
}
@Bean
public Binding ttlBinding(){
    return BindingBuilder.bind(ttlQueue()).to(ttlExchange()).with("ttl");
}

Send a message, but do not specify a TTL:

@Test
public void testTTLQueue() {
    // 创建消息
    String message = "hello, ttl queue";
    // 消息ID,需要封装到CorrelationData中
    CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());
    // 发送消息
    rabbitTemplate.convertAndSend("ttl.direct", "ttl", message, correlationData);
    // 记录日志
    log.debug("发送消息成功");
}

Log of sent messages:

Check the log of received messages:

 

Because the TTL value of the queue is 10000ms, which is 10 seconds. You can see that the time difference between message sending and receiving is exactly 10 seconds.

2.2.3. When sending a message, set TTL

When sending a message, you can also specify a TTL:

@Test
public void testTTLMsg() {
    // 创建消息
    Message message = MessageBuilder
        .withBody("hello, ttl message".getBytes(StandardCharsets.UTF_8))
        .setExpiration("5000")
        .build();
    // 消息ID,需要封装到CorrelationData中
    CorrelationData correlationData = new CorrelationData(UUID.randomUUID().toString());
    // 发送消息
    rabbitTemplate.convertAndSend("ttl.direct", "ttl", message, correlationData);
    log.debug("发送消息成功");
}

Check the sent message log:

Receive message log:

 

This time, the delay between sending and receiving is only 5 seconds. Note that when TTL is set for both the queue and the message, any one that expires will become a dead letter.

2.2.4. Summary

What are the two ways of message timeout?

  • Set the ttl attribute for the queue. Messages that exceed the ttl time after entering the queue will become dead letters.

  • Set the ttl attribute for the message. When the queue receives the message and exceeds the ttl time, it will become a dead letter.

How to send a message 20 seconds before the consumer receives the message?

  • Specify a dead-letter switch for the message's destination queue

  • Bind the queue that the consumer listens to to the dead letter exchange

  • When sending a message, set the timeout for the message to 20 seconds

2.3. Delay queue

Using TTL combined with the dead letter switch, we achieve the effect of delaying the receipt of the message by the consumer after the message is sent. This message mode is called the Delay Queue mode.

Usage scenarios for delay queues include:

  • Delay sending SMS

  • If the user places an order and does not pay within 15 minutes, it will be automatically canceled.

  • Schedule a work meeting and automatically notify all participants after 20 minutes

Because there is a lot of demand for delay queues, RabbitMQ officially launched a plug-in that natively supports the delay queue effect.

This plug-in is the DelayExchange plug-in. Refer to RabbitMQ’s plug-in list page:Community Plugins — RabbitMQ

For usage, please refer to the official website address:Scheduling Messages with RabbitMQ | RabbitMQ - Blog

2.3.1.Install the DelayExchange plug-in

https://blog.csdn.net/m0_66755326/article/details/134957270

2.3.2.DelayExchange principle

DelayExchange requires an exchange to be declared as delayed type. When we send a message to delayExchange, the process is as follows:

  • receive messages

  • Determine whether the message has the x-delay attribute

  • If there is an x-delay attribute, it means that it is a delayed message. It is persisted to the hard disk and the x-delay value is read as the delay time.

  • Return routing not found result to message sender

  • After the x-delay time expires, re-deliver the message to the specified queue

2.3.3. Using DelayExchange

The use of the plug-in is also very simple: declare a switch. The type of the switch can be any type. You only need to set the delayed attribute to true, and then declare the queue to be bound to it.

1) Declare the DelayExchange switch

Based on annotation method (recommended):

 It can also be based on @Bean:

2) Send message

When sending a message, be sure to carry the x-delay attribute to specify the delay time:

 

2.3.4. Summary

What are the steps to use the delay queue plug-in?

•Declare a switch and add the delayed attribute to true

•When sending a message, add the x-delay header, the value is the timeout period

3. Lazy queue

3.1. Message accumulation problem

When the speed of the producer sending messages exceeds the speed of the consumer processing the messages, messages will accumulate in the queue until the queue reaches the upper limit of message storage. Messages sent after that will become dead letters and may be discarded. This is a message accumulation problem.

 

There are two ways to solve message accumulation:

  • Add more consumers and increase consumption speed. This is the work queue mode we talked about before.

  • Expand the queue capacity and increase the accumulation limit

To increase the queue capacity, it is obviously not possible to store messages in memory.

3.2. Lazy queue

Starting from version 3.6.0 of RabbitMQ, the concept of Lazy Queues, which is a lazy queue, has been added. The characteristics of lazy queue are as follows:

  • After receiving the message, it is stored directly in the disk instead of in the memory.

  • When the consumer wants to consume the message, it reads it from the disk and loads it into the memory.

  • Supports millions of message storage

3.2.1. Set lazy-queue based on the command line

To set a queue as a lazy queue, you only need to specify the x-queue-mode attribute as lazy when declaring the queue. A running queue can be modified to a lazy queue via the command line:

rabbitmqctl set_policy Lazy "^lazy-queue$" '{"queue-mode":"lazy"}' --apply-to queues  

Command interpretation:

  • rabbitmqctl: Command line tool for RabbitMQ

  • set_policy:Add a strategy

  • Lazy: Policy name, can be customized

  • "^lazy-queue$": Use regular expression to match the name of the queue

  • '{"queue-mode":"lazy"}':Set the queue mode to lazy mode

  • --apply-to queues: The target of the policy is all queues

3.2.2. Declaring lazy-queue based on @Bean

3.2.3. Declare LazyQueue based on @RabbitListener

3.3. Summary

Solution to message stacking problem?

  • Bind multiple consumers to the queue to increase consumption speed

  • Using lazy queue, you can save more messages in mq

What are the advantages of lazy queue?

  • Based on disk storage, high message limit

  • There is no intermittent page-out, and the performance is relatively stable.

What are the disadvantages of lazy queues?

  • Based on disk storage, message timeliness will be reduced

  • Performance is limited by disk IO

4.MQ cluster

4.1.Cluster classification

RabbitMQ is written based on the Erlang language, and Erlang is a concurrency-oriented language that naturally supports cluster mode. RabbitMQ clusters have two modes:

Ordinary cluster: It is a distributed cluster that disperses the queues to various nodes in the cluster, thereby improving the concurrency capability of the entire cluster. .

Mirror cluster: It is a master-slave cluster. Based on the ordinary cluster, the master-slave backup function is added to improve the cluster's performance. Data availability.

Although the mirror cluster supports master-slave, master-slave synchronization is not strongly consistent, and there may be a risk of data loss in some cases. Therefore, after version 3.8 of RabbitMQ, new features were introduced: Arbitration Queue to replace the mirror cluster. The bottom layer uses the Raft protocol to ensure master-slave data consistency. .

4.2. Ordinary cluster

4.2.1. Cluster structure and characteristics

Ordinary clusters, or classic clusters, have the following characteristics:

  • Some data will be shared among various nodes in the cluster, including: switch and queue meta information. Does not include messages in the queue.

  • When accessing a node in the cluster, if the queue is not on that node, it will be passed from the node where the data is to the current node and returned.

  • If the node where the queue is located is down, the messages in the queue will be lost.

The structure is as shown in the figure:

 

4.2.2.Deployment

https://blog.csdn.net/m0_66755326/article/details/134957270

4.3. Mirror cluster

4.3.1. Cluster structure and characteristics

Mirror cluster: essentially a master-slave mode, with the following characteristics:

  • The switches, queues, and messages in the queues will be synchronously backed up between the mirror nodes of each mq.

  • The node that creates the queue is called the queue'sprimary node,The other nodes backed up to are called the queue's< a i=3>Mirrornode.

  • The master node of one queue may be the mirror node of another queue

  • All operations are completed by the master node and then synchronized to the mirror node.

  • After the master goes down, the mirror node will be replaced by the new master.

The structure is as shown in the figure:

 

4.3.2.Deployment

https://blog.csdn.net/m0_66755326/article/details/134957270

4.4. Arbitration queue

4.4.1. Cluster characteristics

Arbitration queue: Arbitration queue is a new feature only available after version 3.8. It is used to replace the mirror queue and has the following characteristics:

  • Like the mirror queue, both are in master-slave mode and support master-slave data synchronization.

  • Very simple to use, no complicated configuration

  • Master-slave synchronization is based on Raft protocol and is strongly consistent

4.4.2.Deployment

https://blog.csdn.net/m0_66755326/article/details/134957270

4.4.3.Java code creates arbitration queue

@Bean
public Queue quorumQueue() {
    return QueueBuilder
        .durable("quorum.queue") // 持久化
        .quorum() // 仲裁队列
        .build();
}

 

4.4.4.SpringAMQP connects to MQ cluster

Note that address is used here instead of host and port.

spring:
  rabbitmq:
    addresses: 192.168.150.140:8071, 192.168.150.140:8072, 192.168.150.140:8073
    username: itcast
    password: 123321
    virtual-host: /

Guess you like

Origin blog.csdn.net/m0_66755326/article/details/134956746