Past recommendation
[sharding-sphere] - 01 SQL Routing
[Nacos source code analysis] - 02 Obtaining the configuration process
[Java Concurrent Programming] - 03 MESI, memory barrier
[Spring source code]- 11 Programmatic transactions of Spring AOP
【Programming Development】- 01 Log Framework
overview
SimpleMessageListenerContainerIt is a consumer tool class encapsulated and implemented on the native basis. This class is very powerful and can realize: monitoring single or multiple queues, automatic startup, automatic declaration, and it also supports dynamic configuration, such as dynamically adding listening queues and dynamically adjusting the number springof rabbitmqconcurrency apiWait, basically RabbitMQthis class can satisfy the consumption scene. For example @RabbitListener, the middle and bottom implementations in cloud-stream StreamListenerare all based on this class, so understanding SimpleMessageListenerContainerthe principle is very important to understand spring rabbitmqthe consumption model in the middle.
basic use
1. SimpleMessageListenerContainer#addQueueNames()The method can add the listening queue when it is running, and removeQueueNames()the method can remove the listening queue when it is running;
2. Post processorsetAfterReceivePostProcessors()
//后置处理器，接收到的消息都添加了Header请求头
container.setAfterReceivePostProcessors(message -> {
message.getMessageProperties().getHeaders().put("desc",10);
return message;
});
container.setMessageListener((MessageListener) message -> {
System.out.println("====接收到消息=====");
System.out.println(message.getMessageProperties());
System.out.println(new String(message.getBody()));
});
3. Set the Consumer_tag and Arguments of the consumer: container.setConsumerTagStrategy can set the Consumer_tag of the consumer, and container.setConsumerArguments can set the Arguments of the consumer
container.setConsumerTagStrategy(queue -> "order_queue_"+(++count));
//设置消费者的Arguments
Map<String, Object> args = new HashMap<>();
args.put("module","订单模块");
args.put("fun","发送消息");
container.setConsumerArguments(args);
4. setConcurrentConsumers sets concurrent consumers: setConcurrentConsumers sets multiple concurrent consumers to consume together, and supports dynamic modification at runtime. setMaxConcurrentConsumers sets the maximum number of concurrent consumers.
@Bean
public SimpleMessageListenerContainer messageListenerContainer(ConnectionFactory connectionFactory){
SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
container.setConnectionFactory(connectionFactory);
container.setQueueNames("zhihao.miao.order");
container.setConcurrentConsumers(5);
container.setMaxConcurrentConsumers(10);
container.setMessageListener((MessageListener) message -> {
System.out.println("====接收到消息=====");
System.out.println(message.getMessageProperties());
System.out.println(new String(message.getBody()));
});
return container;
}
core principle
API structure
SimpleMessageListenerContainerThe class structure is as follows:
Source code analysis
method entry
SimpleMessageListenerContainerThe entry point for class startup is start()the method, which is located AbstractMessageListenerContainerin the class:
public void start() {
//如果已启动，则什么也不执行，直接退出
if (isRunning()) {
return;
}
//initialized是否执行初始化，没有则执行afterPropertiesSet()方法进行初始化，执行完成后initialized设置成true
if (!this.initialized) {
synchronized (this.lifecycleMonitor) {
if (!this.initialized) {
afterPropertiesSet();
}
}
}
try {
logger.debug("Starting Rabbit listener container.");
//验证RabbitAdmin，mismatchedQueuesFatal=true时，spring context中RabbitAdmin数量不能大于1
configureAdminIfNeeded();
//执行RabbitAdmin#initialize方法，spring context中注入的exchanges, queues and bindings执行声明式创建
/*
总结一下，我们发现，要想自动创建队列，SimpleMessageListenerContainer需要满足这么两点:
mismatchedQueuesFatal属性设置为true
autoDeclare属性也设置为true
*/
checkMismatchedQueues();
//启动核心
doStart();
}
catch (Exception ex) {
throw convertRabbitAccessException(ex);
}
finally {
this.lazyLoad = false;
}
}
SimpleMessageListenerContainer#doStartmethod:
protected void doStart() {
Assert.state(!this.consumerBatchEnabled || getMessageListener() instanceof BatchMessageListener
|| getMessageListener() instanceof ChannelAwareBatchMessageListener,
"When setting 'consumerBatchEnabled' to true, the listener must support batching");
//如果MessageListener是ListenerContainerAware，则进行expectedQueueNames校验
checkListenerContainerAware();
//调用父类doStart()方法，主要是active和running都设置成true
super.doStart();
synchronized (this.consumersMonitor) {
if (this.consumers != null) {
throw new IllegalStateException("A stopped container should not have consumers");
}
/*
创建BlockingQueueConsumer类型consumer，每个concurrentConsumers并发对应创建一个对象，并存储到Set<BlockingQueueConsumer> consumers集合中，
返回值就是创建consumer对象个数，具体创建逻辑见：SimpleMessageListenerContainer#createBlockingQueueConsumer，主要注意下prefetchCount计算：
int actualPrefetchCount = getPrefetchCount() > this.batchSize ? getPrefetchCount() : this.batchSize;即如果prefetchCount大于batchSize，则其就是实际值，否则prefetchCount等于batchSize值
*/
int newConsumers = initializeConsumers();
if (this.consumers == null) {
logger.info("Consumers were initialized and then cleared " +
"(presumably the container was stopped concurrently)");
return;
}
if (newConsumers <= 0) {
if (logger.isInfoEnabled()) {
logger.info("Consumers are already running");
}
return;
}
/*
每个并发对应一个BlockingQueueConsumer对象，这里将每个BlockingQueueConsumer对象封装成AsyncMessageProcessingConsumer，这样可以丢到线程池中异步执行
*/
Set<AsyncMessageProcessingConsumer> processors = new HashSet<AsyncMessageProcessingConsumer>();
for (BlockingQueueConsumer consumer : this.consumers) {
//将BlockingQueueConsumer对象封装成AsyncMessageProcessingConsumer进行异步执行
AsyncMessageProcessingConsumer processor = new AsyncMessageProcessingConsumer(consumer);
//存储到processors集合中
processors.add(processor);
//将AsyncMessageProcessingConsumer丢到线程池中执行
getTaskExecutor().execute(processor);
if (getApplicationEventPublisher() != null) {
//事件发送
getApplicationEventPublisher().publishEvent(new AsyncConsumerStartedEvent(this, consumer));
}
}
//判断启动过程中是否存在异常
waitForConsumersToStart(processors);
}
}
The above code is roughly logical: BlockingQueueConsumerthe object can be regarded as consumer, and then packaged into AsyncMessageProcessingConsumeran asynchronous task and thrown into the thread pool to run.
asynchronous task
BlockingQueueConsumerThe types analyzed above consumerwill be encapsulated into AsyncMessageProcessingConsumerasynchronous tasks and thrown into the thread pool to run. The following mainly analyzes what is done when the asynchronous task is executed. The logic is in SimpleMessageListenerContainer.AsyncMessageProcessingConsumer#runthe method, which mainly does the following things:
1. Listening queue judgment
//BlockingQueueConsumer.getQueueCount() < 1，表示当前consumer没有设置任何监听队列，则没必要启动
if (this.consumer.getQueueCount() < 1) {
if (logger.isDebugEnabled()) {
logger.debug("Consumer stopping; no queues for " + this.consumer);
}
SimpleMessageListenerContainer.this.cancellationLock.release(this.consumer);
if (getApplicationEventPublisher() != null) {
getApplicationEventPublisher().publishEvent(
new AsyncConsumerStoppedEvent(SimpleMessageListenerContainer.this, this.consumer));
}
this.start.countDown();
return;
}
2. Core logic
try {
initialize();
while (isActive(this.consumer) || this.consumer.hasDelivery() || !this.consumer.cancelled()) {
mainLoop();
}
}
catch (InterruptedException e) {
省略。。。
}
catch (QueuesNotAvailableException ex) {
省略。。。
}
省略。。。
AsyncMessageProcessingConsumer#runThe core logic of execution is that in the above try statement, initialize()the initialization method is executed first, and then mainLoop()the method is executed in an infinite loop.
initialization
The runtime analyzed above AsyncMessageProcessingConsumerperforms two key operations: initialize()an initialization operation and mainLoop()an infinite loop.
First of all, let's look at initialize()what the initialization operation mainly does:
Call
attemptDeclarations()methods for declarativeexchange,queue, andbindingscreation, which are mainlyRabbitAdmin#initializerealized by executing methods;Call
BlockingQueueConsumer#startthe method, which mainly completesRabbit Brokerthe interaction with the command:1.
passiveDeclarations()The method checks whether the listening queue exists:channel.queueDeclarePassive(queueName), and will eventuallyRabbit Brokersendqueue.declarecommands to and setpassive=true, as shown in the figure below:
2. setQosAndreateConsumers()The method is used for client flow control Qosand message subscription
a. Qos flow control: channel.basicQos(this.prefetchCount), will eventually Rabbit Brokersend basic.qosinstructions to the server, and set the parameters of prefetch-size, , prefetch-countand global, as shown in the figure below:
b. consumeFromQueue()The method will use channel.basicConsumethe method to subscribe to the message, and finally Rabbit Brokersend basic.consumean instruction to the user, and specify queuethe name of the subscribed message and other parameter messages (as shown in the figure below). Note: SimpleMessageListenerContainermultiple listening queues may be set, and BlockingQueueConsumereach listening queue will be sent to the Broker. A basic.consumesubscription directive, and uses the same one channel:
BrokerIn the method of sending instructions to com.rabbitmq.client.impl.AMQCommand#transmit(), you can monitor code breakpoints at the following red boxes:
response processing
In the above analysis of initialize()the initialization operation, the client Brokersends instructions to the server, which is equivalent to telling the server: I am ready, and if there is a message in the listening queue, you can push it to me. Next, let’s analyze the basic.qosmessage push process.basic.consumeBroker
Rabbit BrokerAfter receiving Basic.consumethe command, it will feed back the command to the client Basic.consume-ok, indicating that the server is ready to push the message to the client, and then Basic.Deliverpush the message to the client through the command type. A message corresponds to a Deliverfeedback, and the client receives the server and returns it. After the instruction type, ChannelN#processAsyncthe method is judged and processed, which amqp-clientdepends on the class in the package:
processDelivery()
the method for processing:
protected void processDelivery(Command command, Basic.Deliver method) {
Basic.Deliver m = method;
//根据Deliver的consumerTag获取到InternalConsumer对象，因为一个Channel上可能存在多个consumer，需要找到Broker是针对哪个consumer进行的响应
Consumer callback = _consumers.get(m.getConsumerTag());
if (callback == null) {
if (defaultConsumer == null) {
// No handler set. We should blow up as this message
// needs acking, just dropping it is not enough. See bug
// 22587 for discussion.
throw new IllegalStateException("Unsolicited delivery -" +
" see Channel.setDefaultConsumer to handle this" +
" case.");
}
else {
callback = defaultConsumer;
}
}
Envelope envelope = new Envelope(m.getDeliveryTag(),
m.getRedelivered(),
m.getExchange(),
m.getRoutingKey());
try {
// call metricsCollector before the dispatching (which is async anyway)
// this way, the message is inside the stats before it is handled
// in case a manual ack in the callback, the stats will be able to record the ack
metricsCollector.consumedMessage(this, m.getDeliveryTag(), m.getConsumerTag());
this.dispatcher.handleDelivery(callback,
m.getConsumerTag(),
envelope,
(BasicProperties) command.getContentHeader(),
command.getContentBody());
} catch (WorkPoolFullException e) {
// couldn't enqueue in work pool, propagating
throw e;
} catch (Throwable ex) {
getConnection().getExceptionHandler().handleConsumerException(this,
ex,
callback,
m.getConsumerTag(),
"handleDelivery");
}
}
processDelivery()The general flow of processing Brokerthe returned Delivermessage:
Consumer callback = _consumers.get(m.getConsumerTag()): According toDelivertheconsumerTagobtainedInternalConsumerobject, becauseChannelthere may be more than one on oneconsumer, it is necessary to find outBrokerwhichconsumerresponse is madePackaged
DeliverasEnvelope:Envelope envelope = new Envelope(m.getDeliveryTag(), m.getRedelivered(), m.getExchange(), m.getRoutingKey());metricsCollector.consumedMessage(this, m.getDeliveryTag(), m.getConsumerTag()): statistical data processingCall
ConsumerDispatcher#handleDelivery, which will create a task and throw it into the thread pool for execution. Task: hand over the data to specificconsumerprocessing, that is, callInternalConsumer#handleDeliverythis.dispatcher.handleDelivery(callback, m.getConsumerTag(), envelope, (BasicProperties) command.getContentHeader(), command.getContentBody());InternalConsumer#handleDelivery()Method: PutBrokerthe returnedDeliverdata intoBlockingQueueConsumer.queue:
BlockingQueueConsumer.this.queue.put(new Delivery(consumerTag, envelope, properties, body, this.queueName));
Therefore, if ListenerContaineryou listen to multiple queues, BlockingQueueConsumerthere are multiple queues in the middle, one on InternalConsumereach InternalConsumermapping , and all the lower ones share the same one .BrokerBlockingQueueConsumerInternalConsumerqueue
business processing
The above analyzes the message subscription and the message data pushed by the Broker will be cached in BlockingQueueConsumerthe object queuequeue. Next, we will analyze the process queuefrom extracting messages to delivering them to the user's business logic. This requires AsyncMessageProcessingConsumer#runanother very important operation in the analysis method: the infinite loop mainLoopoperation, which mainly completes the extraction of message data queuefrom it and then passes it to the user logic after a series of operations MessageListener.
private void mainLoop() throws Exception { // NOSONAR Exception
try {
boolean receivedOk = receiveAndExecute(this.consumer); // At least one message received
if (SimpleMessageListenerContainer.this.maxConcurrentConsumers != null) {
checkAdjust(receivedOk);
}
long idleEventInterval = getIdleEventInterval();
if (idleEventInterval > 0) {
if (receivedOk) {
updateLastReceive();
}
else {
long now = System.currentTimeMillis();
long lastAlertAt = SimpleMessageListenerContainer.this.lastNoMessageAlert.get();
long lastReceive = getLastReceive();
if (now > lastReceive + idleEventInterval
&& now > lastAlertAt + idleEventInterval
&& SimpleMessageListenerContainer.this.lastNoMessageAlert
.compareAndSet(lastAlertAt, now)) {
publishIdleContainerEvent(now - lastReceive);
}
}
}
}
catch (ListenerExecutionFailedException ex) {
// Continue to process, otherwise re-throw
if (ex.getCause() instanceof NoSuchMethodException) {
throw new FatalListenerExecutionException("Invalid listener", ex);
}
}
catch (AmqpRejectAndDontRequeueException rejectEx) {
/*
* These will normally be wrapped by an LEFE if thrown by the
* listener, but we will also honor it if thrown by an
* error handler.
*/
}
}
Follow up doReceiveAndExecute():
private boolean doReceiveAndExecute(BlockingQueueConsumer consumer) throws Exception { //NOSONAR
Channel channel = consumer.getChannel();
List<Message> messages = null;
long deliveryTag = 0;
//batchSize默认是1，用于指定一次从queue中提取消息数量
for (int i = 0; i < this.batchSize; i++) {
logger.trace("Waiting for message from consumer.");
Message message = consumer.nextMessage(this.receiveTimeout);
if (message == null) {
break;
}
//是否批处理
if (this.consumerBatchEnabled) {
Collection<MessagePostProcessor> afterReceivePostProcessors = getAfterReceivePostProcessors();
if (afterReceivePostProcessors != null) {
Message original = message;
deliveryTag = message.getMessageProperties().getDeliveryTag();
for (MessagePostProcessor processor : getAfterReceivePostProcessors()) {
message = processor.postProcessMessage(message);
if (message == null) {
channel.basicAck(deliveryTag, false);
if (this.logger.isDebugEnabled()) {
this.logger.debug(
"Message Post Processor returned 'null', discarding message " + original);
}
break;
}
}
}
if (message != null) {
if (messages == null) {
messages = new ArrayList<>(this.batchSize);
}
if (isDeBatchingEnabled() && getBatchingStrategy().canDebatch(message.getMessageProperties())) {
final List<Message> messageList = messages;
getBatchingStrategy().deBatch(message, fragment -> messageList.add(fragment));
}
else {
messages.add(message);
}
}
}
else {
messages = debatch(message);
if (messages != null) {
break;
}
try {
//执行MessageListener
executeListener(channel, message);
}
catch (ImmediateAcknowledgeAmqpException e) {
if (this.logger.isDebugEnabled()) {
this.logger.debug("User requested ack for failed delivery '"
+ e.getMessage() + "': "
+ message.getMessageProperties().getDeliveryTag());
}
break;
}
catch (Exception ex) {
if (causeChainHasImmediateAcknowledgeAmqpException(ex)) {
if (this.logger.isDebugEnabled()) {
this.logger.debug("User requested ack for failed delivery: "
+ message.getMessageProperties().getDeliveryTag());
}
break;
}
if (getTransactionManager() != null) {
if (getTransactionAttribute().rollbackOn(ex)) {
RabbitResourceHolder resourceHolder = (RabbitResourceHolder) TransactionSynchronizationManager
.getResource(getConnectionFactory());
if (resourceHolder != null) {
consumer.clearDeliveryTags();
}
else {
/*
* If we don't actually have a transaction, we have to roll back
* manually. See prepareHolderForRollback().
*/
consumer.rollbackOnExceptionIfNecessary(ex);
}
throw ex; // encompassing transaction will handle the rollback.
}
else {
if (this.logger.isDebugEnabled()) {
this.logger.debug("No rollback for " + ex);
}
break;
}
}
else {
consumer.rollbackOnExceptionIfNecessary(ex);
throw ex;
}
}
}
}
if (messages != null) {
executeWithList(channel, messages, deliveryTag, consumer);
}
return consumer.commitIfNecessary(isChannelLocallyTransacted());
}
Summarize
The above SimpleMessageListenerContaineranalysis of the core source code is rather boring and not intuitive. To sum up, the core is the AsyncMessageProcessingConsumer#runtwo operations in the method: initialize()and infinite loop mainLoop().
initialize
initialize()The method is mainly completed: tell the need to monitor the queue information through the instruction method Rabbit Broker, push the message to the client Brokerthrough the instruction when there is message data in the monitoring queue , after the client receives the instruction, send it according to the distribution ( ) , and then put it into To the queue of the object it belongs to , its logic can be seen in the figure below:DeliverDeliverconsumerTagdispatcherconsumerconsumerBlockingQueueConsumerqueue
BlockingQueueConsumer, AsyncMessageProcessingConsumer, listening queue and other relationships:
1. BlockingQueueConsumerIt is equivalent to a logical consumer, which is encapsulated into AsyncMessageProcessingConsumeran asynchronous task and then thrown into the thread pool to run. The thread pool can be SimpleMessageListenerContainer#setTaskExecutorconfigured through customization, so BlockingQueueConsumerit can be regarded as a separate thread running, and corresponds to one Channel;
2. SimpleMessageListenerContainerYou can listen to multiple queue messages, and each queue will create an InternalConsumerobject for the concept Brokerof mapping. consumerThey share the same one channel, that is, channelthere are multiple consumerones, and they are consumerTagdistinguished. In addition, Brokerthe push message is also based on consumerTagidentification Which one will be pushed to consumerfor processing?
Cases such as:
container.setQueueNames("test01", "test02");
container.setConcurrentConsumers(3);
container.setConsumerTagStrategy(queue -> "consumer idx:"+consumerIdx.getAndIncrement());
concurrentConsumersa. Create the corresponding number according to the number of concurrency BlockingQueueConsumer, then encapsulate it AsyncMessageProcessingConsumer, and then assign a thread to execute it. Here it is set to 3, so there will be 3 threads running AsyncMessageProcessingConsumer, each AsyncMessageProcessingConsumercorresponding to a channel, so 3 channels will be created, in the Web UI You can see the corresponding channel above:
b. Each listening queue creates a InternalConsumermapping with the Broker's consumer. There are two listening queues, so there will be 2 consumers under each channel:
AsyncMessageProcessingConsumerHow to subscribe:
a. First send Basic.Qosan instruction to agree on the message push rate;
b. Then send Basic.Consumean instruction to tell Brokerthe client to start subscribing to the message on the queue, and to bring it consumerTagon. Because there may be multiple listening queues, multiple instructions channelmay be sent on the same one. When pushing the message, it needs to find the corresponding processing ;Basic.ConsumeBrokerchannelconsumerTagconsumer
c. Broker pushes the message to the client through the Deliver instruction type. After receiving the message, the client finds the corresponding consumer according to the consumerTag and hands it over to process, that is, distributes the dispatcher;
d. The consumer here corresponds to that InternalConsumerits processing logic is to put it into BlockingQueueConsumerthe message queue of the object where it is located queue;
mainLoop
The messages pushed by the Broker are put into the message queue queue of the BlockingQueueConsumer object, and then the messages are extracted from the queue for business processing. The logic is shown in the figure below:
a. AsyncMessageProcessingConsumerIf it is thrown into the thread pool for execution, it corresponds to a thread;
b. This thread will always execute mainLoop()the method in a loop;
c. mainLoop()In the method, queue中the message will be extracted from, and batchSizethe number of messages extracted each time is determined, and finally called back MessageListener, so as to transfer the message to the business logic for processing;
d. Note: All AsyncMessageProcessingConsumershare the same MessageListenerobject, and the object state should pay attention to thread safety issues;
overall process
Long press the QR code to identify attention