How much do you know about MQ? (Dry goods sharing part 2)

Introduction

This article sorts out the author's MQ knowledge, starting from the basic knowledge of message middleware, and after having the basic knowledge, makes a detailed analysis of the mainstream message middleware on the market, including RabbitMQ, RocketMQ, Kafka, Pulsar, and finally compares them horizontally These mainstream messaging middleware. This article is the second in a series of articles.

RocketMQ

basic concept

Tag

Tag (label) can be seen as a subtopic, which is a second-level type of message, used to provide users with additional flexibility. Using tags, messages with different purposes in the same business module can be identified with the same Topic but different Tags. For example, transaction messages can be divided into: transaction creation messages, transaction completion messages, etc. A message may not have a Tag. Tags help keep your code clean and coherent, and also help with the query system provided by RocketMQ.

Group

In RocketMQ, the concept of subscribers is embodied by consumer groups. Each consumer group consumes a complete message in the topic, and the consumption progress between different consumer groups is not affected by each other. That is to say, if a message has been consumed by Consumer Group1, it will be consumed by Consumer Group2. A consumer group contains multiple consumers. Consumers in the same group are in a competitive consumption relationship, and each consumer is responsible for consuming part of the messages in the group. By default, if a message is consumed by consumer Consumer1, other consumers in the same group will not receive this message again.

Offset

In the consumption process of Topic, since messages need to be consumed by different groups multiple times, the consumed messages will not be deleted immediately, which requires RocketMQ to maintain a consumption position on each queue for each consumption group ( Consumer Offset), the previous messages at this position have been consumed, and the subsequent messages have not been consumed. Every time a message is successfully consumed, the consumption position will be increased by one. It can also be said that Queue is an array of unlimited length, and Offset is the subscript.

RocketMQ Architecture

RabbitMQ similarly has a production stage, a storage stage, and a consumption stage. Compared with the RabbitMQ architecture, a NameServer cluster is added, and the horizontal expansion capability is better. The reference Kafka design also has NIO, PageCache, sequential read and write, and zero-copy skills. The throughput of a single machine is at the level of 100,000, and the horizontal expansion capability is strong. It is officially stated that the cluster can carry a trillion-level throughput.

In the storage phase, you can configure Broker parameters that prioritize reliability to avoid losing messages due to downtime. Simply put, synchronization should be used in scenarios where reliability is prioritized.

1. As long as the message is persisted to the CommitLog (log file), even if the Broker is down, the unconsumed message can be recovered and consumed again.

2. Broker's brushing mechanism: synchronous and asynchronous brushing, no matter which kind of brushing can guarantee that the message will be stored in Pagecache (in memory), but synchronous disk is more reliable, it is the data that is sent after the Producer sends the message After persisting to the disk, the response is returned to the Producer.

Broker ensures high availability through the master-slave mode. Broker supports Master and Slave synchronous replication, Master and Slave asynchronous replication mode. The producer's message is sent to the Master, but consumption can be consumed from the Master or from the Slave. The synchronous replication mode can ensure that even if the Master goes down, the message must be backed up in the Slave, ensuring that the message will not be lost.

In the Consumer configuration file, it is not necessary to set whether to read from the Master or the Slave. When the Master is unavailable or busy, the Consumer's read request will be automatically switched to the Slave. With the mechanism of automatic switching of consumers, when a machine in the role of master fails, consumers can still read messages from slaves without affecting consumers' reading of messages, which realizes high availability of reading.

How to achieve high availability of writing at the sending end? When creating a Topic, create multiple Message Queues of the Topic on multiple Broker groups (the same Broker name, different BrokerId machines form a Broker group), so that when the Master of the Broker group is unavailable, the Master of other groups is still available, and the Producer Messages can still be sent.

RocketMQ under this architecture does not support automatic conversion of Slave to Master. If the machine resources are insufficient and it is necessary to convert Slave to Master, you must manually stop the Slave-colored Broker, change the configuration file, and start the Broker with the new configuration file. Therefore, this problem becomes difficult in high-availability scenarios, so it is necessary to introduce the implementation of distributed algorithms to pursue CAP. However, in practice, CA cannot be satisfied at the same time. In Internet scenarios, it is mostly based on the time BASE theory, which is prioritized. AP, try to satisfy C as much as possible. RocketMQ introduces Dledger that implements the Raft algorithm. It has election capabilities and master-slave switching. The architecture topology is as follows:

Paxos algorithm is often heard among distributed algorithms, but because Paxos algorithm is difficult to understand and difficult to implement, it is not very popular in the industry. Then came the new distributed algorithm Raft, which is easier to understand and implement than Paxos, and it has been very mature in practice, and different languages have implemented it. Dledger is one of the implementations of the Java language, which abstracts away all the content of the algorithm, so that developers only need to deal with business, which greatly reduces the difficulty of use.

transactional message

Producers send messages to the Apache RocketMQ server.
After the Apache RocketMQ server successfully persists the message, it returns an Ack to the producer to confirm that the message has been sent successfully. At this time, the message is marked as "temporarily undeliverable", and the message in this state is a semi-transactional message.
The producer starts executing local transaction logic.
The producer submits the second confirmation result (Commit or Rollback) to the server according to the execution result of the local transaction, and the processing logic of the server after receiving the confirmation result is as follows:

The result of the second confirmation is Commit: the server marks the half-transaction message as deliverable and delivers it to the consumer.

The result of the second confirmation is Rollback: the server will roll back the transaction and will not deliver the half-transaction message to the consumer.
In the special case of network disconnection or producer application restart, if the server does not receive the second confirmation result submitted by the sender, or the second confirmation result received by the server is in the Unknown state, after a fixed period of time, the service The terminal will initiate a message checkback to the message producer, that is, any producer instance in the producer cluster. For the interval time and maximum number of checkbacks on the server side, please refer to [ Parameter Limits ]( https://rocketmq.apache.org/zh/docs/introduction/03limits/ ).
After the producer receives the message checkback, it needs to check the final result of the local transaction execution of the corresponding message.
The producer submits a second confirmation based on the checked final state of the local transaction, and the server still processes the half-transaction message according to step 4.

Transaction message lifecycle

Initialization : The semi-transactional message is constructed and initialized by the producer, and is ready to be sent to the server.

Transactions to be submitted : Semi-transactional messages are sent to the server. Unlike ordinary messages, they will not be directly persisted by the server, but will be stored separately in the transactional storage system, and wait for the second stage of the local transaction to return the execution result. Submit again. At this point the message is not visible to downstream consumers.

Message rollback : In the second stage, if the transaction execution result is clearly rollback, the server will roll back the half-transaction message, and the transaction message process will terminate.

Submit for consumption : In the second stage, if the transaction execution result is clearly submitted, the server will re-store the semi-transactional message in the common storage system. At this time, the message is visible to downstream consumers, waiting to be acquired and consumed by consumers.

Consuming : The process in which the message is acquired by the consumer and processed according to the consumer's local business logic. At this time, the server will wait for the consumer to complete the consumption and submit the consumption result. If no response is received from the consumer after a certain period of time, Apache RocketMQ will retry the message. For details, see Consumption Retries.

Consumption submission : The consumer completes the consumption process and submits the consumption result to the server, and the server marks that the current message has been processed (including consumption success and failure). Apache RocketMQ supports retaining all messages by default. At this time, the message data will not be deleted immediately, but the logical mark has been consumed. Before the message is deleted when the storage time expires or the storage space is insufficient, the consumer can still go back and consume the message again.

Message deletion : Apache RocketMQ rolls and clears the earliest message data according to the message preservation mechanism, and deletes the message from the physical file. See Message Store and [ Cleanup Mechanism ]( https://rocketmq.apache.org/en/docs/featureBehavior/11messagestorepolicy/ ) for more information.

RocketMQ new development

In the past, "separation" was often a compromise of technical realization, but now "combination" is the real demand of users. RocketMQ 5.0 expands diversified indexes based on unified Commitlog, including time index, million queue index, transaction index, KV index, batch index, logical queue and other technologies. In the scene, it also supports product capabilities such as RabbitMQ, Kafka, MQTT, and edge lightweight computing, and strives to realize the extended support of "messages, events, and streams". Cloud native is the mainstream.

More information can be found on the official website [ Apache RocketMQ ] ( https://rocketmq.apache.org/zh/ ).

Kafka

Kafka is a distributed system consisting of servers and clients that communicate over the high-performance TCP network protocol. It can be deployed on bare-metal hardware, virtual machines, and containers in on-premises and cloud environments.

Servers : Kafka runs as a cluster of one or more servers, which can span multiple data centers or cloud regions. Some of these servers form the storage layer, called proxies. Other servers run Kafka Connect to continuously import and export data as a stream of events to integrate Kafka with your existing systems such as relational databases and other Kafka clusters. To allow you to implement mission-critical use cases, Kafka clusters are highly scalable and fault-tolerant: if any one of these servers fails, the others will take over to ensure continuous operation without any data loss.

Clients : They allow you to write distributed applications and microservices that can read, write, and process streams of events in parallel and at scale in a fault-tolerant manner, even in the presence of network issues or machine failures. Kafka ships with a few such clients, augmented by dozens of clients contributed by the Kafka community: clients are available for Java and Scala, including higher-level Kafka Streams libraries for Go, Python, C /C++ and many other programming languages and REST APIs.

architecture

Similar to the previous two MQs, there are production phases, storage phases, and consumption phases. Compared with RocketMQ, the registration center here uses Zookeeper, and many events in Kafka depend on ZK, metadata management, registration of each role, heartbeat, election, State maintenance, the roles here include Boker, Topic, Partition, consumer group, etc.

Therefore, there will also be a problem of excessive ZK Watch event pressure. A large number of ZK node events are blocked in the queue, resulting in spin locks, causing CPU to rise, and a large number of event objects occupying a large amount of memory.

The Controller in the figure is the concept of the Kakfa server Broker. There are multiple Broker clusters, but only one Broker can play the role of the controller; once a Broker becomes a Controller, it is used for the following rights: complete cluster member management, topic Maintenance and partition management, such as cluster Broker information, Topic maintenance, Partition maintenance, partition election ISR, synchronization of meta information to other Brokers, etc.

storage

Topic is a logical concept, while Partition is a physical concept, that is, a Topic is divided into multiple Partitions, and each Partition corresponds to a Log file.

.log file : A file that stores message data.
.index file : Index file, which records the position of a message in the log file.
.snapshot file : Records the latest offset of the producer.
.timeindex time index file : The timestamp of the indexed message in the current log segment file is added after version 0.10.0, which is used to quickly find the displacement value of a specific message according to the timestamp, and optimize Kafka to read historical messages slowly The problem. In order to ensure the monotonous increase of the timestamp, log.message.timestamp.type can be set to logApendTime, but CreateTime cannot guarantee the message writing time.

The above figure shows the storage situation of three Brokers, two Topics, and two Partition Brokers. It can be extended to imagine that the storage situation of a million-level Topic will be very complicated.

Rebalnce problem

In order to solve the problems caused by the strong reliance on Zookeeper for Rebalance, Kafka introduces the Coordinator mechanism.

First of all, the scenarios that trigger the Rebalance (rebalance) operation are currently divided into the following categories: the number of consumers in the consumer group changes, including:

A new customer joins
Consumers who go offline due to downtime, including real downtime, or long-term GC or network delays that cause consumers to fail to send heartbeats to GroupCoordinator within the timeout period, will also be considered offline.
Some consumers voluntarily exit the consumer group (send a LeaveGroupRequest request) For example, the client calls the unsubscrible() method to cancel the subscription to some topics
The GroupCoordinator node corresponding to the consumer group has changed.
The topic subscribed by the consumer group changes (increases or decreases) or the number of topic partitions changes.
Node expansion

For more information, please check Kafka official website [ Apache Kafka ]( https://kafka.apache.org/ )

Pulsar

At the highest level, a Pulsar instance consists of one or more Pulsar clusters. Clusters within an instance can replicate data between them.

In a Pulsar cluster:

One or more Brokers process and load balance incoming messages from producers, dispatch messages to consumers, communicate with Pulsar configuration storage to handle various coordination tasks, store messages in BookKeeper instances (aka bookies), depend on Cluster-specific ZooKeeper clusters are used for certain tasks and so on.

A BookKeeper cluster of one or more Bookies handles the persistent storage of messages.

A ZooKeeper cluster specific to this cluster handles coordination tasks between Pulsar clusters.

The following figure shows a Pulsar cluster:

Pulsar uses Apache BookKeeper as persistent storage, Broker holds BookKeeper client, and sends unconfirmed messages to BookKeeper for storage.

BookKeeper is a distributed WAL (Write Ahead Log) system. Pulsar uses BookKeeper with the following advantages:

Multiple Ledgers can be created for Topic: Ledger is an append-only data structure, and there is only one Writer, which is responsible for writing to multiple BookKeeper storage nodes (that is, Bookies). Ledger entries are copied to multiple Bookies;
Broker can create, close and delete Ledger, and can also append content to Ledger;
After the Ledger is closed, it can only be opened in a read-only state, unless data is explicitly written or closed because the Writer hangs;
Ledger can only be written by the Writer process, so there will be no conflicts in writing, so the writing efficiency is very high. If the Writer hangs up, Ledger will start the recovery process to determine the final state of Ledger and the last submitted log, so that all Ledger processes will read the same content afterwards;
In addition to saving message data, Cursors are also saved, which is where the consumer subscribes to and consumes. In this way, the Ledger can be deleted after all the Cursors have consumed the messages of a Ledger, so that the regular rolling of Ledgers can be realized from scratch.

peer-to-peer

As can be seen from the architecture diagram, Broker nodes do not save data, and all Broker nodes are equal. If a Broker goes down, no data will be lost, only the Topic it serves needs to be migrated to a new Broker.

Broker's Topic has multiple logical partitions, and each partition has multiple Segments.

When Writer writes data, it first selects Bookies, such as Segment1 in the figure. Select Bookie1, Bookie2, Bookie4, and then write them concurrently. In this way, the three nodes do not have a master-slave relationship, and the coordination is completely dependent on the Writer, so they are also equal.

Expansion and expansion

When encountering high-traffic scenarios such as Double Eleven, consumers must be added.

At this time, because Broker does not store any data, you can easily add Broker. A Broker cluster will have one or more Brokers for message load balancing. When a new broker joins, the traffic will be automatically migrated from the stressed broker.

For BookKeeper, if the storage requirements become higher, such as storing 2 copies before and now needing to store 4 copies, Bookies can be expanded independently at this time without considering Broker. Because the nodes are equal, the segments of the previous nodes are stacked neatly, and there is no need to move data when adding a new node. Writer will perceive new nodes and choose to use them first.

fault tolerance mechanism

For Broker, because no data is saved, if the node goes down, it means that the client is disconnected, and it is enough to reconnect to other Brokers.

For BookKeeper, multiple copies are kept and these copies are all peers. Because there is no master-slave relationship, when a node goes down, it does not need to recover immediately. There is a thread in the background that checks the data backup of the down node for recovery.

When encountering high-traffic scenarios such as Double Eleven, consumers must be added.

Pulsar can use multi-tenancy to manage large clusters. Tenants of Pulsar can be distributed across clusters, and each tenant can have a separate authentication and authorization mechanism. A tenant is also the administrative unit for storage quotas, message TTLs, and quarantine policies.

In terms of docking with other components or ecology, Pulsar can support a variety of message protocols, which is very convenient for the first access to MQ of the stock system and switching MQ.

More information can be found on Pulsar official website [ Apache Pulsar ]( https://pulsar.apache.org/ )

Compared

This picture is excerpted from "Face Slag Counterattack: RocketMQ Twenty-Three Questions"

There is no Pulsar information in this figure. According to the pressure test report seen on the Internet, the throughput of Pulsar is about twice that of Kafka, and the latency performance is much lower than that of Kafka. The I/O isolation of Pulsar is significantly better than Kafka. For a more detailed comparison of Pulsar and Kafka, please refer to StreamNative's article "Pulsar and Kafka Benchmark Test: Accurate Analysis of Pulsar Performance (Full Version)". As a commercial company of Apache Pulsar, StreamNative has relatively reliable data and results.

Advanced

As the saying goes, the best way to learn is to take questions to find answers, pick up more fruits on the way, increase experience points, and quickly level up. Many people recommend the Feynman learning method, which uses teaching instead of learning, learning according to the standards that can be taught to others, and finally producing teaching content for the purpose of learning a knowledge, so that they can learn efficiently. In my opinion, this is very similar to the OKR tool for performance appraisal, setting key results for the project, what should be done to achieve success? How to do it? And I am writing this article in the practice of Feynman learning method.

Therefore, here I give a few questions, and readers can take questions to find answers according to their own interests and hobbies.

How to ensure the availability/reliability/no loss of messages?
How to deal with the problem of message duplication?
How are sequential messages implemented?
How to deal with message backlog?
How to implement distributed message transactions? half news?
How to implement message filtering?

If you usually think of too many questions and don't know which one to look at first, then you should figure out why you need to learn these knowledge points, and which question will benefit you the most.

References:

Official website document address:
RabbitMQ official website document: https://www.rabbitmq.com/documentation.html

Apache RocketMQ official website documentation: https://rocketmq.apache.org/zh/docs/

Apache Kafka official website documentation: https://Kafka.apache.org/documentation/

Apache Pulsar official website documentation: https://pulsar.apache.org/docs/

Past

recommend

" About MQ, how much do you know?" (One of the dry goods sharing)》

" Tencent Cloud Message Queue Product Updates in March "

" Tencent Cloud Microservice Products March Product Updates "

" Dry Goods: Exploration of Kafka's High Reliability and High Performance Principles "

" Solving Heterogeneous System Integration Problems, Furong Bank Doing It "

" Apache Pulsar Technology Series - Pulsar Overview "

" To solve the three structural problems of innovative business, CNR Shopping has used this key strategy correctly "

" Detailed Apache Pulsar message life cycle "

"Based on Tencent Cloud Microservice Engine (TSE), it is easy to realize full-link grayscale release on the cloud "

Scan the QR code below to follow this official account,

Learn more about microservices and message queues!

Unlock a lot of goose factory surroundings!

Click the original text to see more

Message queue RocketMQ

information!

Click to see you look the best

This article is shared from the WeChat public account - Tencent Cloud Middleware (gh_6ea1bc2dd5fd).
If there is any infringement, please contact [email protected] to delete it.
This article participates in the " OSC Source Creation Program ". You are welcome to join in and share it.

How much do you know about MQ? (Dry goods sharing part 2)

Guess you like