Source code analysis RocketMQ set different tag with a consumer group, news subscription failures

Background

The project team using Ali RocketMQ, set a different tag subscription relationship with a consumer group, message loss problems, they publish and subscribe principle from rocketmq news source research, and analyze the cause of the problem.

Official Description

• Tell the user: the same consumer group, you must subscribe to the same relationship
• why? It did not say that! Only to find the answer from source

Recurring problem

1. 1 Start consumer, the consumer group group1, subscribe topicA news, tag set tag1 || tag2
2. 2 Start consumer, the consumer group also group1, topicA also subscribe to the news, but the tag is set to tag3
3. Start the producer, the producer transmits containing Tag1, each 10 tag2, tag3 message
4. 1 consumer has not received any news that consumers receive a 2-part message

First Conclusion

• The same consumer groups, setting different tag, consumers will start after the cover tag before consumers start setting
• tag determines the conditions for message filtering, and client service through two layers of filtration, and finally start only after the consumers can receive a partial message

Rationale

How to save the message

CommitLog

• Save all the topic of the original message
• CommitLog into multiple files, each up to 1G default
• Each record comprising: a message length and message text (message body, property, etc. UID)
• Because the length of each message is inconsistent, the length of each record is also inconsistent commitLog

ConsumerQueue

• Save a Queue under a Topic of index information
• Each record includes: the hash value of the offset commitLog message, message size, a message tag
• Each record has a fixed length 20byte
• After sending the message producer, save to commitLog, then made asynchronously topic + queue message corresponding to the corresponding piece of index ConsumerQueue
• Hash (tag) The third part is an important basis for server-side message filtering

How consumer subscription message

Sign up for information

• consumer subscription, the subscription will be registered to the information to the server
• Map information is stored subscription class, Key to Topic , value mainly tag
• subVersion take the current time .

  The key here is the topic, subVersion version number, which are two very critical! Useful back to!

Pull message and filtered

• When the pull message, first get the subscription relationship from the server to obtain the hash tag collection codeSet
• Then acquires one record from ConsumerQueue, determines whether the hashCode codeSet recorded in order to achieve the purpose of filtering the message, decide whether to send the message to the consumer
• In a word: Tag determine whether the message is sent to the client

Message Filtering

Server filter

• Filter: hash value tag filter
• advantage:
  • Reduce unnecessary traffic news occupation
• Disadvantages:
  • Hash there is a conflict, the filter is not entirely accurate

Client Filter

• The server filter does not exist accuracy, precision filter client again
• Customer FILTERING: string tag values ​​do comparison. Unequal does not return to the consumer

The reason summary

• The same consumer group subscription relationship, stored in the Map RebalanceImpl class. key for the topic
• After different consumers start, followed by registration subscription relationship, because the tag is not the same, resulting in Map of tag in the same topic is covered . For example: 1 consumer subscription tag1, 2 consumer subscription tag2. Finally, the map only save tag2.
• The core filter Yes Yes Tag, Tag is updated filter conditions are changed . Tag2 returns only the service end message filtering
• After the client receives the message, and filtered again. 1 first start consumer subscription tagA, but the server returns tag2, so consumers can not receive any messages 1. Consumers can receive half of the message 2 (trunked mode, assuming equal distribution message, the other half give Tag2)

Source code analysis

Subscribe to relational data structures

Consumers subscribe to 1 to start the registration of the relationship

After two starts covering consumer subscription relationship

ConsumerQueue taken when the service end filter Hash (tag)

Contrast Hash message (tag) and saved prior to the subscription relationship

Client Filter

Reproduced in: https: //juejin.im/post/5cfa7065f265da1ba647dddd