RocketMQ consumer core configuration and core knowledge

A, RocketMQ4.X consumer core configuration

• consumeFromWhere configuration (in some cases failure: Reference https://blog.csdn.net/a417930422/article/details/83585397 ) This configuration is not changed substantially, to use the default settings.
  • CONSUME_FROM_FIRST_OFFSET: Initial start spending from the message queue head, that history messages (also stored in the broker's) total consumption again, followed by subsequent re-start progress of the last consumer start spending.
  • CONSUME_FROM_LAST_OFFSET: default policy, for the first time starting from the end of the queue most consumer and skip the history of news, followed by subsequent re-start progress of the last consumer start spending.
  • CONSUME_FROM_TIMESTAMP: start spending from a point in time, the default is half an hour ago, followed by subsequent re-start progress of the last consumer start spending.
• allocateMessageQueueStrategy: load balancing strategy algorithm that consumers assign to queue algorithm
  • The default value is the average distribution AllocateMessageQueueAveragely i.e. modulo
• offsetStore: news consumption schedule memory offsetStore There are two strategies: LocalFileOffsetStore and RemoteBrokerOffsetStore 
  • Broadcast mode uses LocalFileOffsetStore default, cluster model is used by default RemoteBrokerOffsetStore
• consumeThreadMax: The maximum number of consumer thread pool
• consumeThreadMin: minimum number of consumer thread pool
• pullBatchSize: When consumers pull message to the broker, once pulled how many. Optional
• consumeMessageBatchMaxSize: how many messages during a single one-time consumer consumption, volume consumer interface to be useful, optional
• messageModel: consumer spending patterns
  • CLUSTERING: Cluster mode (the default configuration)
  • BROADCASTING: Broadcast mode

 Two, RocketMQ consumer side processing clustered and broadcast mode

Parity Topic next queue will affect the amount of consumption Customer number inside

• If the queue is four, eight messages, the node 4 will be two of the consumer, if unequal, the load balancing is allocated unevenly.
• If the number of consumer instances more than the total number of message queue, then the extra consumer instances can not be assigned to the queue, the message will not be able to consume, it can not play a role in sharing the load, so it is necessary to control so that the total queue It is equal to the number greater than the number of the consumer.

Cluster mode (the default):

• Message Consumer instances the average share of consumption producers sent
• Examples: order message, usually consumed only once (marked as a ConsumerGroup group of consumers with messages will not be repeated consumption)

Broadcast mode:

• Consumption message broadcast mode: to deliver a message is Broker consume each Consumer, a plurality of Consumer consumption message is broadcast ConsumerGroup temporarily useless consumption.
• Examples: Group announcement, everyone needs to consume the news

How the switching mode: setMessageModel ()

Three, RocketMQ inside Tag action and message filtration principle

A Message is only one Tag, Tag is a secondary classification. Consumer end and into the filter Broker end filtration.

• Broker-end filtration, reduce unnecessary message transmission network, increasing the burden on the broker
• Consumer end of the filter, the filter can be based on business needs, but added a lot of unnecessary message transmission

* Generally monitor, or specify Tag, || operation, SLQ92, FilterServer the like;

• Tag high performance, simple logic
• SQL92 almost performance, support for complex logic (only supported in PushConsumer) MessageSelector.bySql
  • Syntax:>, <, =, IS NULL, AND, OR, NOT, etc., sql where subsequent syntax to (mostly)

 Producers

@RequestMapping("/api/v1/pay_cb")
public Object callback( String tag, String amount) throws Exception {

    Message the Message = new new the Message (JmsConfig.TOPIC, Tag, "" , tag.getBytes ());
     // set properties, for filtering sql 
    message.putUserProperty ( " AMOUNT " , AMOUNT);
    
    SendResult sendResult =  payProducer.getProducer().send(message);
    System.out.printf("发送结果=%s, sendResult=%s \n", sendResult.getSendStatus(), sendResult.toString());
    return new HashMap<>();
}

consumer

package net.xdclass.xdclassmq.jms;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.MessageSelector;
import org.apache.rocketmq.client.consumer.listener.*;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.springframework.stereotype.Component;

import java.io.UnsupportedEncodingException;
import java.util.List;

@Component
public class PayConsumer {


    private DefaultMQPushConsumer consumer;

    private String consumerGroup = "pay_consumer_group";

    public  PayConsumer() throws MQClientException {

        consumer = new DefaultMQPushConsumer(consumerGroup);
        consumer.setNamesrvAddr(JmsConfig.NAME_SERVER);
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
        // default is the cluster approach, it can be changed to broadcast, but broadcast does not support retry 
        consumer.setMessageModel (MessageModel.CLUSTERING);
         // multi-label subscription
         // Consumer.subscribe (JmsConfig.TOPIC, "order_pay order_finish || || order_create ");

        // filter messages according to the syntax sql 
        Consumer.subscribe (JmsConfig.TOPIC, MessageSelector.bySql ( "AMOUNT>. 5" ));

        consumer.registerMessageListener( new MessageListenerConcurrently() {

            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
                MessageExt msg = msgs.get(0);

                try {
                System.out.printf("%s 2 Receive New Messages: %s %n", Thread.currentThread().getName(), new String(msgs.get(0).getBody()));

                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;

            } catch (Exception e) {
                System.out.println ( "Consumer exception" );
                e.printStackTrace ();
                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
            }
            }
        });

        consumer.start();
        System.out.println("consumer start ...");
    }

}

Note: Consumers subscribe to the relationship to be consistent, otherwise chaos will consume even the message is lost. Subscribe consistent relationship: the relationship between the subscription Topic Tag and composition, with a group name, Topic, and subscribe to the Tag must be the same.

In principle be filtered MessageTag end Broker: traversal message queue stored in the tag hashcode message tag and whether subscription delivery of the same, is not the same is skipped, the transmission is to meet Consumer, is stored in the consumer queue corresponding hashCode, comparison is after receiving the filtered message Consumer matching operation will be carried out, but the comparison is not true message tag hashcode; hashcode by comparison.

• hashcode consume queue using memory length, space-saving
• Filtration not accessed commit log, the filter can be efficiently
• If the hash conflict exists, Consumer end can be confirmed again

Recommendation: single responsibility, multiple queues; If you want to use multiple Tag, you can use the sql expression, but not recommended.

Common mistakes:

The broker does not support consumer to filter message by SQL92

Solve: broker.conf which configuration is as follows
enablePropertyFilter=true

After the Notes, modify to restart Broker
master node configuration: vim conf / 2m-2s-async / broker-a.properties
slave node configuration: vim conf / 2m-2s-async / broker-as.properties

 

Four, PushConsumer / PullConsumer news consumption patterns

Push and Pull advantages and disadvantages

• Push: real-time high; but increases the server load, different consumer end capabilities, if Push push too fast, the consumer side there will be a lot of problems
• Pull: Pull Consumers take messages from the Server side, the initiative in the consumer side, good controllability; but a bad set time interval, the interval is too short, empty request, waste of resources; interval is too long, then the message is not timely deal with
• Long Polling: Client Server Request Broker is the end of time, Broker will maintain the current connection for some time, default is 15s, if there is this period of time a message arrives, then immediately returned to the Consumer; no news, then, over 15s, return empty, then re-request; Consumer the initiative, even if a large number of messages Broker does not proactively pushed to the Consumer. Disadvantages: the need to maintain the server request Consumer, and will take up resources needed number of client connections controlled, otherwise there will be a bunch of connection

PushConsumer nature is long in rotation

• After receiving the message system to automatically process messages and offset, if there is a new Consumer automatically added to do load balancing,
• At the end of the broker may be opened by a long polling longPollingEnable = true
• Although the push, but the code which makes heavy use of pull, because in rotation using a long way to achieve the effect of push, pull both real-time and some, another push of
• Elegant Off: The main is to release resources and save Offset, call the shutdown () to reference @ PostConstruct, @ PreDestroy

PullConsumer needs its own maintenance Offset (refer to the official example)

• Official source package example path: org.apache.rocketmq.example.simple.PullConsumer
• Get MessageQueue traversal
• Customer maintenance Offset, users need to use local storage Offset, storage memory, disk, databases, etc.
• Different processing status message FOUND, NO_NEW_MSG, OFFSET_ILLRGL, NO_MATCHED_MSG, 4 states
• High flexibility, strong controllability, but coding complexity will be high
• Elegant Off: The main is to release resources and save Offset, save yourself a good program required Offset, especially when exception handling