Spring Cloud Alibaba study notes (8) - RocketMQ terminology and concepts

Topic

A set of message type, RocketMQ basic unit subscriptions

Deployment Architecture

Name Server

Name Server provides routing information for the producer and the consumer.
Relatively speaking, nameserver very high stability. There are two reasons:

• nameserver independently of each other, there is no communication relationship with each other, a single nameserver hang up, does not affect other nameserver, even if all hang up, does not affect the business system. no status
• nameserver does not have frequent read and write, so the performance overhead is very small, high stability.

Broker Server

Proxy server, the message transfer roles, is responsible for storing messages, forward messages

Message Model

Producer

Message producers, the role of the producer is to send a message to the MQ, message producers themselves can be produced both as to read the text information and the like.
External interfaces can also be provided by an external application to call interface, and then send the message to the producer will receive MQ.

Relationship with NameServer

• connection
  • And individual producers who NameServer holding a long connection, polling the topic configuration information, if the hang NameServer producer NameServer will automatically connect to the next, until a connection is made available, and to automatically reconnect.
• Heartbeat
  • No heartbeat connection
• Polling Time
  • By default, the producers every 30 seconds to get all the latest topic of the queue situation from NameServer, which means that if a broker is down, the producers take up to 30 seconds to perceive, in the meantime, sent a message to the broker failure. This time is determined by the pollNameServerInteval DefaultMQProducer parameter can be configured manually.

Relationship with Borker

• connection
  • All individual producers and producer associated with the broker to maintain a long connection.
• Heartbeat
  • By default, the producer sends a heartbeat every 30 seconds for all Broker, which time is determined by heartbeatBrokerInterval DefaultMQProducer parameter can be configured manually. broker every 10 seconds (this time can not be changed), is connected to scan all still alive, if a connection over 2 minutes (the current time and the last update time difference than 2 minutes, this time can not be changed) the heartbeat data is not transmitted, then close the connection.
• disconnect
  • Remove producer information on broker

Broker

Broker receiving a message from the producer, the consumer storage and pull message requests prepared.

Relationship with NameServer

• connection
  • And all individual broker to maintain a long connection nameserver
• Heartbeat
  • Heartbeat intervals: every 30 seconds (this time can not be changed) to send a heartbeat to all nameserver, heart comprising their own configuration information topic.
  • Heartbeat Timeout: nameserver every 10 seconds (this time can not be changed), scan all still alive broker connections, if a connection within 2 minutes (the current time and the last update time difference over two minutes, this time can not be changed) did not transmitting the heartbeat data is disconnected.
• disconnect
  • Timing: broker hang; heartbeat timeout lead the initiative to close the connection nameserver
  • Action: Once disconnected, nameserver will perceive immediately update the correspondence between topic and queue, but does not notify producers and consumers

Load Balancing

• A topic distributed across multiple broker, a broker may be configured to a plurality of topic, which is many to many relationship.
• If a large amount of news topic, it should give more than a few configuration queue? What does that mean, as much as possible and distributed on different broker, a broker of reducing stress.
• The amount of message topic are relatively uniform, if more queues on a broker, the broker pressure increases.

Consumer

News consumers, in simple terms, application messages on the MQ consumer is the consumer, as to whether the message processing logic, memory or directly to the database depends on your business needs.

Relationship with NameServer

• connection
  • And a single consumer nameserver holding a long connection, polling the topic configuration information, if the nameserver hang up, the consumer is connected automatically to the next nameserver, until a connection is made available, and to automatically reconnect.
• Heartbeat
  • Not associated with heartbeat
• Polling Time
  • By default, consumers every 30 seconds to get all the latest topic of the queue situation from nameserver, which means that if a broker is down, the client take up to 30 seconds to perceive. This time is determined by the pollNameServerInteval DefaultMQPushConsumer parameter can be configured manually.

Relationship with Borker

• connection
  • All the individual consumer and the consumer associated broker to maintain a long connection.
• Heartbeat
  • By default, the consumer sends a heartbeat every 30 seconds for all Broker, which time is determined by heartbeatBrokerInterval DefaultMQPushConsumer parameter can be configured manually. broker every 10 seconds (this time can not be changed), is connected to scan all still alive, if a connection over 2 minutes (the current time and the last update time difference than 2 minutes, this time can not be changed) the heartbeat data is not transmitted, it closes the connection and notify all consumers of the consumer groups, consumer groups within the redistribution queues continue to spend.
• disconnect
  • Timing: Consumers hang up; the heartbeat timeout lead the initiative to close the connection broker
  • Action: Once disconnected, broker immediately perceived, and to notify all consumers of the consumer group, within the group of consumers continue to spend reallocate queue

Load Balancing

• Cluster-consumption mode, a consumer cluster together multiple machines consume a topic of multiple queues, a queue will be consumed by a consumer.
• If a consumer hang up, within the group of other consumers will take over hang of consumers continue to spend.

Consumer mechanism

• Local queue
  • Consumers uninterrupted pull messages from the broker, the message queue pulled to local, local consumption and local consumption of threads message queue, just an asynchronous process, pull the thread does not wait for local consumption of threads, this model in real time is very high. Consumers have the protection of a local queue, so the local message queue can not be infinite, this may take up a lot of memory, local queue size pullThresholdForQueue property controls DefaultMQPushConsumer by default 1000, can be set manually.
• Polling interval
  • How often pull a message thread pulling once? Intervals controlled DefaultMQPushConsumer pullInterval property by default is 0, it can be set manually.
• Message of consumption
  • Every listener to accept local queue of messages is how many? This parameter controls the consumeMessageBatchMaxSize DefaultMQPushConsumer property, the default is 1, can be set manually.

Storage consumption schedule

• The progress of the consumption of the respective storage queue corresponding to the intervals Broker, the time persistConsumerOffsetInterval DefaultMQPushConsumer controlled by the property, the default is 5 seconds, can be set manually.

Connections

• If there is a topic on a broker three queues, a consumer spending these three queues, then the consumer and the broker has several connection?
• A connected consumer unit associated with the queue, the consumer is connected only with the relevant broker, in fact, all queues of messages consumers will pull into the local task queue, one by one pull, the pull is complete, turn pulling tasks put to the tail, and then perform the next task of pulling

Group

Producer Group

Producer group, is simply the same type of a plurality of producers transmit messages called a producer group.

Consumer Group

Consumer groups, producers and similar consumer multiple instances of the same class of consumer messages constitute a consumer group (cluster).

Tag

Tags can be considered a further refinement of the Topic. Usually the introduction of different uses tags to mark the same message by the service module.
RocketMQ support to fight to the topic tag when sending the same message, although a topic of management logic is the same. But when the consumption topic1, if you subscribe to the time specified is tagA, then the message will not be delivered tagB.

Message

Message is the payload of the message. Message must specify a topic, the equivalent of an address to send a letter.
Message An optional tag disposed so that the end consumer can filter messages based on the tag.
You can also add additional key-value pairs, such as a business you need to find the key messages on the broker, easy to diagnose problems in the development process.

Message Queue

Acronym queue or Q, the physical management of message units. Topic will have a number of Q. If Topic simultaneously created on different Broker, the broker has a number of different Q, physically stored message will fall on different nodes Broker, horizontal scaling ability.

Offset

In RocketMQ, there are many concepts of offset. But usually we only care about the client's exposure to the offset. Generally, we are not a specific words, refers to the following logic Message Queue offset.

Consumer Offset

Consumer Group for marking on a logical Message Queue, where the message consumer. Note: From the source point of view, this value is offset + 1 is the latest piece of news consumption, so in fact this value is stored [next pull, then where to start pulling the offset].

reference

https://www.jianshu.com/p/3938d990a8de
https://www.cnblogs.com/qdhxhz/p/11094624.html