Click "Middleware Interest Circle" above and select "Set as Star"
Be a positive person, the harder you work, the luckier you get!
Before officially entering the study of RocketMQ, I think it is necessary to sort out the core concepts of RocketMQ and lay a solid foundation for everyone to learn RocketMQ.
1. RocketMQ deployment architecture
Nameserver
The Nameserver cluster, the routing registration center of the topic, provides routing services for the client according to the topic, thereby guiding the client to send messages to the Broker. Nodes between Nameservers do not communicate. The eventual consistency of routing information in the nameserver cluster data consistency.
Broker
The message storage server is divided into two roles: Master and Slave. The above figure shows the deployment architecture of 2 masters and 2 slaves. In RocketMQ, the master service undertakes read and write operations, and the slave server acts as a backup. When the master server is under pressure , the slave server can undertake the read service (message consumption). All Brokers, including the Slave server, will send heartbeat packets to the Nameserver every 30s, and the heartbeat packets will contain the routing information of all topics existing on the Broker.
Client
Message client, including Producer (message sender) and Consumer (consumer). The client will only connect to one nameserver at the same time, and will try to connect to the other one only when the connection is abnormal. The client initiates a topic routing information query to the Nameserver every 30s.
Reminder: Nameserver stores topic routing information in memory, and the place to persist topic routing information is in Broker, that is, ${ROCKETMQ_HOME}/store/config/topics.json.
After the RocketMQ4.5.0 version, a multi-copy mechanism was introduced, that is, a replication group (ms) can evolve into a replication group based on the raft protocol. The raft protocol is used inside the replication group to ensure the strong consistency of broker node data. This deployment architecture is widely used in the financial industry. Used more.
2. Message subscription model
The message consumption mode in RocketMQ adopts the publish and subscribe mode.
topic: a collection of a type of message, the sender sends a type of message to a topic, for example, the order module sends the order to order_topic, and when the user logs in, the login event is sent to user_login_topic.
consumegroup: message consumption group, a "group" of consumption units. The consumption group first needs to subscribe to the topic that needs to be consumed when it is started. A topic can be subscribed by multiple consumer groups, and a consumer group can also subscribe to multiple topics. A consumer group has multiple consumers.
The terminology is a bit boring and obscure to explain, so I will illustrate it with an example.
For example, we are developing an order system with a subsystem: order-service-app. In this project, a consumer group order_consumer will be created to subscribe to order_topic. Based on distributed deployment, the deployment of order-service-app is as follows:
2.1 Consumption Mode
How do these three consumers divide their work to consume the messages in order_topic together?
Broadcast mode and cluster mode are supported in RocketMQ.
Broadcast mode : All consumers in a consumer group will each process each message in the topic, usually used to flush the memory cache.
Cluster mode : All consumers in a consumer group consume messages in a topic together, that is, division of labor and cooperation, one consumer consumes a part of data, and starts load balancing.
The cluster mode is a very common mode, which conforms to the basic concept of distributed architecture, that is, horizontal expansion. If the current consumers cannot process messages quickly and in a timely manner, they can expand horizontally by increasing the number of consumers to quickly improve consumption capacity and deal with crowding in time. Pressed message.
2.2 Consumption Queue Load Algorithm and Rebalancing Mechanism
In the cluster mode, how do consumers distribute messages?
For example, in the above example, order_topic has 16 queues, and how does a consumer group with 3 consumers allocate the queues.
在MQ领域有一个不成文的约定：同一个消费者同一时间可以分配多个队列，但一个队列同一时间只会分配给一个消费者。
RocketMQ提供了众多的队列负载算法，其中最常用的两种平均分配算法。
AllocateMessageQueueAveragely
平均分配
AllocateMessageQueueAveragelyByCircle
轮流平均分配
为了说明这两种分配算法的分配规则，现在对16个队列，进行编号，用q0~q15表示，消费者用c0～c2表示。
AllocateMessageQueueAveragely分配算法的队列负载机制如下：
c0：q0 q1 q2 q3 q4 q5
c1: q6 q7 q8 q9 q10
c2: q11 q12 q13 q14 q15
其算法的特点是用总数除以消费者个数，余数按消费者顺序分配给消费者，故c0会多分配一个队列，而且队列分配是连续的。
AllocateMessageQueueAveragelyByCircle分配算法的队列负载机制如下：
c0：q0 q3 q6 q9 q12 q15
c1: q1 q4 q7 q10 q13
c2: q2 q5 q8 q11 q14
该分配算法的特点就是轮流一个一个分配。
温馨提示：如果topic的队列个数小于消费者的个数，那有些消费者无法分配到消息。在RocketMQ中一个topic的队列数直接决定了最大消费者的个数，但topic队列个数的增加对RocketMQ的性能不会产生影响。
在实际过程中，对主题进行扩容(增加队列个数)或者对消费者进行扩容、缩容是一件非常寻常的事情，那如果新增一个消费者，该消费者消费哪些队列呢？这就涉及到消息消费队列的重新分配，即消费队列重平衡机制。
In the RocketMQ client, the number of all queues and consumers of the current topic will be queried every 20s, and the queue load algorithm will be used for reallocation, and then compared with the previous allocation result. If there is a change, the queue will be re-allocated. Assignment; ignored if no change has occurred.
For example, the allocation algorithm adopted is shown in the figure below. Now add a consumer c3. What is the distribution of the queue?
c0：q0 q1 q2 q3
c1: q4 q5 q6 q7
c2: q8 q9 q10 q11
c3: q12 q13 q14 q15
The whole process above is done by RocketMQ without application intervention. The general approach is to discard the queue that was originally allocated to itself but does not belong to this time, and the newly allocated queue creates a new pull task.
2.3 Consumption progress
After a consumer consumes a message, it needs to record the consumption location, so that when the consumer restarts, it continues to process new messages from the last consumption location. In RocketMQ, the storage of message consumption sites is in units of consumption groups.
In cluster mode , message consumption progress is stored on the broker side.
不能识别此Latex公式:
{ ROCKETMQ_HOME }/store/config/consumerOffset.json 是其具体的存储文件，其中内容截图如下：
可见消费进度的Key为：topic@consumeGroup，然后每一个队列一个偏移量。
广播模式的消费进度文件存储在用户的主目录，默认文件全路劲名：
2.4 Consumption Model
RocketMQ provides two consumption models: concurrent consumption and sequential consumption.
Concurrent consumption : For messages in a queue, each consumer will create a thread pool internally to process messages in the queue in multiple threads, that is, messages with larger offsets may be consumed first than messages with smaller offsets.
Sequential consumption : In a certain scenario, such as the MySQL binlog scenario, messages need to be consumed in order. A queue-based sequential consumption model is provided in RocketMQ, that is, although consumers in a consumption group will create a multi-thread, they will be locked for the same Queue.
Reminder: In the concurrent consumption model, if the message consumption fails, it will be retried 16 times by default, and the interval time between each time is different; in the case of sequential consumption, if a message fails to be consumed, it will continue to be consumed until the consumption is successful. Therefore, in the process of sequential consumption, the application needs to distinguish between system exceptions and business exceptions. If it is an exception caused by non-compliance with business rules, the consumption cannot be successfully retried no matter how many times. At this time, an alarm mechanism must be used, and human intervention must be carried out in time. , otherwise there will be a backlog of consumption.
3. Transaction message
Transaction messages are not to solve distributed transactions, but to provide consistency between message sending and business storage. The implementation principle is the specific application of a distributed transaction. Please see the following example:
Warm reminder, the main purpose of this section is to let everyone know the concept of each term. Due to the use of transaction messages, it will be introduced in detail in the subsequent articles of this column.
4. Timed messages
The open source version of RocketMQ does not currently support arbitrary precision timing messages. The so-called timing message is to send the message to the broker, but the consumer will not consume it immediately, but will not be consumed by the consumer until the specified delay time.
RocketMQ currently supports a specified level of delay, and its delay level is as follows:
1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
5. Message filtering
Message filtering means that the consumer can filter the messages in a topic according to certain conditions, that is, only consume messages that meet the filtering conditions under one topic.
RocketMQ's current main filtering mechanisms are tag-based filtering and message attribute-based filtering. Message attribute-based filtering supports SQL92 expressions to filter messages.
6. Summary
The main purpose of this article is to introduce the common terms of RocketMQ, such as nameserver, broker, topic, consumer group, consumer, queue load algorithm, queue rebalancing mechanism, concurrent consumption, sequential consumption, consumption progress storage, timing message, transaction message, message Basic concepts such as filtering lay a solid foundation for the follow-up combat series.
This article is from the author's other masterpiece "RocketMQ Actual Combat and Advancement". The column starts with the usage scenario to introduce how to use RocketMQ, what problems are encountered during the use, how to solve these problems, and why they can be solved in this way, that is, the principle explanation (Figure) interspersed in actual combat . The design idea of the column emphasizes the word actual combat, which aims to make a RocketMQ beginner quickly "fight monsters and upgrade" through the study of this column, combining theory with actual combat, and become a leader in this field.
This article is shared from WeChat public account - middleware interest circle (dingwpmz_zjj).
If there is any infringement, please contact [email protected] to delete it.
This article participates in the " OSC Yuanchuang Project ", you are welcome to join and share with us.