Springboot2.x RabbitMQ achieve integrated current limiting consumers, manual ack confirmation
Foreword
In practice, we project that might accumulate in mq in thousands of messages, if we do not limit, when we open the consumer when all of a sudden thousands of messages all of a sudden shocks over, might cause the server down, or business severe vulnerability, so we need consumers limiting. First of all my springboot version, springBootVersion = '2.2.1.RELEASE'. Other versions of the configuration differences are not large.
Note: In the previous version 2.0, only one MessageListenerContainer-SimpleMessageListenerContainer; a second container --DirectMessageListenerContainer after 2.0, configured as follows:
listener:
#Container where the RabbitMQ consumer dispatches messages to an invoker thread.
type: simple
simple:
acknowledge-mode: manual
prefetch: 1
#Container where the listener is invoked directly on the RabbitMQ consumer thread.
type: direct
direct:
acknowledge-mode: manual
prefetch: 1
SimpleMessageListenerContainer
By default, the listener will start a single container user, the user receives the message from the queue.
In an inspection on the table, you will see a lot of concurrency control properties. The simplest is concurrentConsumers, only the number of users created (fixed) to concurrent processing of the message.
In addition, also added a new attribute maxConcurrentConsumers, the container will dynamically adjusted based on workload concurrency. This work, together with four additional properties: continutiveactivetrigger, startConsumerMinInterval, continutiveidletrigger, stopConsumerMinInterval.
In the default setting, increase consumer work algorithm is as follows:
If you have not yet reached maxConcurrentConsumers, and has 10 consecutive cycles of the user is active and a user since the start has been at least 10 seconds, then start a new user. If a user receives at least one message in txSize *, it is considered that the user is active.
In the default setting, reducing the consumer's algorithm works as follows:
If multiple concurrentConsumers is running, and a consumer detects 10 consecutive timeout (idle), and at least one consumer is stopped before 60 seconds, then the consumer will stop. It depends on receiveTimeout and txSize property. If the user does not receive any message txSize *, it is considered that it is idle. Thus, after the default timeout (one second) and txSize is 4, the idle time of 40 seconds (4 correspond to an idle timeout is detected) a user will be considered stopped.
DirectMessageListenerContainer
Use DirectMessageListenerContainer, you need to ensure ConnectionFactory to configure a task executor, the executor has enough threads to support the required concurrency in all listeners in the container using the ConnectionFactory. The default connection pool size is only 5.
Based concurrency of queues and consumersPerQueue. Each queue for each user using a single channel, the concurrency control rabbit client library; By default, it uses five thread pool; You can configure taskExecutor to provide the desired maximum concurrency.
Compared
SimpleMessageListenerContainer provides the following features, but does not provide DirectMessageListenerContainer:
-
txSize- use SimpleMessageListenerContainer, you can set it to control the number of messages delivered affairs and / or reduce the number of ack, but this may result in increasing the number after the failure of repeated transfer. (With txSize the SimpleMessageListenerContainer, DirectMessageListenerContainer there mesagesPerAck, can be used to reduce ack, but can not be used for transaction - Each message delivery and packaged in a single transaction).
-
maxconcurrentconsumer telescoping spacer and consumer / No automatic retractable trigger -DirectMessageListenerContainer; however, it allows you to programmatically change consumersPerQueue attributes, and adjusting the user accordingly.
However, compared with SimpleMessageListenerContainer, DirectMessageListenerContainer has the following advantages:
- Add and delete queue at run time more efficiently; use SimpleMessageListenerContainer, the entire user thread Restart (all users to cancel and re-create); for DirectMessageListenerContainer, affected users will not be canceled.
Avoiding context switches between RabbitMQ client and consumer threads. - A thread is shared across users, not for SimpleMessageListenerContainer each user has a dedicated thread. However, please refer to the important notes about the connection factory configured in a "threads and asynchronous consumers."
YML 配置
spring:
application:
name: zoo-plus-rabbitmq
rabbitmq:
virtual-host: /
host: localhost
port: 5672
username: guest
password: guest
listener:
type: simple
simple:
acknowledge-mode: manual #采用手动应答
prefetch: 1 #限制每次发送一条数据。
Create a queue:
/**
* 测试队列
*/
@Bean
public Queue testQueue() {
return QueueBuilder.nonDurable("test-queue").build();
}
Producer:
/**
* 发送五条数据，测试消费端必须ack才发送第二条，消费者限流
*/
@GetMapping("ack")
public Resp testAck() {
for (int i = 0; i < 5; i++) {
rabbitTemplate.convertAndSend("test-queue", "测试ack确认模式");
}
return Resp.success("ok", null);
}
consumer:
@RabbitListener(queues = {"test-queue"})
public void testQueue(Message message, Channel channel) throws IOException {
log.info("test-queue消费：" + new String(message.getBody()));
/*
listener:
type: simple
simple:
#采用手动应答
acknowledge-mode: manual
prefetch: 1 #限制每次发送一条数据。
*/
// 采用手动ack，一条条的消费
channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
/*
* 还可以nack
* 第三个参数是否重回队列
*/
// channel.basicNack(message.getMessageProperties().getDeliveryTag(),false,true);
}
From the above test, if we are not configured, all of a sudden five messages on all crowded up, and if we add the configuration, not the consumer side were ack, then the message will unacked, next time you restart the service will once again send this message, until the consumer side ack.
Source address: https: //gitee.com/zoo-plus/springboot-learn/tree/2.x/springboot-middleware/rabbitmq
