Review finishing - MQ

MQ is a message service middleware, based on high-availability distributed cluster technology, and a message system with a consumption model based on a publish-subscribe model. It supports Java, C++, .NET, PHP, and Python, providing asynchronous decoupling, peak-shaving and valley-filling capabilities for distributed application systems, and features such as massive message accumulation, high throughput, and reliable retry. It has the functions of message query, message backtracking (not message recall, nor support for message recall), message track query, and accumulation monitoring and alarm functions.
MQ protocol supports access methods: TCP, HTTP (RESTful style), MQTT. MQ supports public network access, but the availability is low.
MQ application scenarios: distributed transactions, Internet of Things applications, real-time computing (real-time flow of the generated data into the real-time computing engine), synchronization of large-scale caches.
Real-time computing engines generally include: Spark / Storm / EMR / ARMS / BeamRunner.
MQ has management tools: Web console, Open API, mqadmin command set. It has components such as Micro Message Queue (LMQ), RocketMQ Message Queue, Kafka Message Queue, Cross Domain Relay Service (CRS).
The web console provides operations such as message query, message track query, reset consumption site, resource statistics, monitoring and alarming. There are three ways to query messages: According to Message ID (exact query), Message Key (fuzzy query) and Topic query (range query), HTTP messages currently only support two query methods: Message ID and Topic. The message track query only supports TCP and HTTP protocols, and can track the time and place of each relevant node in the entire link from the producer to the consumer. Resetting the consumption point can skip the accumulated messages, that is, you do not want to consume this part of the messages, or you only want to consume messages after a certain point in time (whether these messages have been consumed before). The resource report can collect statistics on the data of message sending and message consumption. It does not support the statistical query of HTTP consumption data. Monitoring alarms are generally used to send SMS messages to alarm recipients after the number of messages accumulated or the delay time exceeds the threshold. If a large number of messages are found, you can check whether the threshold is set too small and cause the accumulation of messages. You can adjust the business code or expand the capacity of consumers. You can use jstack to see if the consuming thread is blocked.
Micro Message Queue (LMQ) is based on the MQTT (Message Queuing Telemetry Transport) protocol, the standard protocol port is 1883, and supports encrypted SSL, WebSocket, and Flash access methods. The important parts of the protocol are mainly divided into: MQ Core Service (responsible for underlying message storage and distribution), MQ private protocol server and MQTT protocol gateway server (responsible for providing services and protocol conversion to clients). The main usage scenarios are: live interaction, Internet of Vehicles, financial payment, instant chat, etc. Protocol related: QoS (Quality of Service) refers to the quality of service of message transmission. It includes three levels of QoS0 (distributed at most once), QoS1 (at least once) and QoS2 (distributed only once). cleanSession identifies whether the client cares about the previous state after establishing a TCP connection (true or false).
MQTT can perform instance management (view information such as message sending and receiving TPS, the number of simultaneous online connections, and the number of subscription relationships, and you can set instance alarms), apply for MQTT Topic, and apply for MQTT Group ID for Topic (a group of nodes with completely consistent logical functions are shared). The group name represents a class of devices with the same function, and must have the read and write permissions of the topic). Signature calculation and signature generation are available.
MQTT can obtain offline messages and actively pull offline messages. The maximum number of messages pulled by the client each time is 30, and the maximum frequency of pull requests is limited to 5 times per second. Offline messages have low priority, and they can be processed in a limited and final way, requiring more real-time.
MQTT can obtain the online and offline events of the client (when the online and offline events are triggered, an online and offline message will be pushed to the back-end MQ, which can be obtained by subscribing to this message). connect (the client is online), disconnect (the client actively disconnects), tcpclean (the actual TCP connection is disconnected). tcpclean represents the real disconnection of the client network layer connection. Please use the tcpclean event to judge the client offline.
MQTT provides access rights to clients through the Token authentication service. The client needs to use the MQTT control packet in the synchronous sending mode and the QoS must be 1 to upload the Token. The client should persist the Token and monitor the notification message of the Token invalidation pushed down by the Proxy. If the Token fails, it must apply again.
For LMQ topics, the maximum length of ClientId is 64 characters, the maximum message size is 64K, the maximum message storage time is 3 days, and the maximum number of topics subscribed by a single client is 30 (topics that exceed the limit will be discarded). Sequentiality is ascending order.
The cross-domain relay service (CRS, cross-domain, realizes service publishing and subscription, and realizes service interoperability of different networks) provides three MQ message sending methods: Reliable synchronous sending (the next message can only be sent after a message response is sent, and the application scenarios are wide. , such as important notification emails, registration SMS notifications, marketing SMS systems), reliable and asynchronous sending (the next message can be sent without waiting for a response, and the application scenario is generally time-consuming and sensitive to RT response services, such as notification transfer after video uploading. Transcoding service, notification push transcoding result after transcoding), One Way (one-way sending, no response is required, time-consuming is very short, and it is used in scenarios that do not require high reliability, such as log collection).
In the MQ message system, resources are divided into message (Message), message producer (Producer), message consumer (Consumer), and message topic (Topic). Producer ID (producer ID), Consumer ID (consumer ID), topic name, must be globally unique.
MQ messages are the carrier of information transmission in message queues. There are generally four types according to types: unordered messages (normal messages, timing messages, transaction messages), global order messages, partition order messages, and Kafka messages. MQ cannot guarantee that messages will not be repeated under abnormal conditions such as network jitter and application processing timeout, but it can guarantee that messages will not be lost. MQ messages can be stored on the server for up to 3 days, and the length of the message body is limited to 256K. The North China 2 region supports 4MB large messages.
The MQ message topic is the first-level classification of the message. The message publisher sends the message to a message topic (Topic), and the message subscriber subscribes to the topic to obtain and consume messages (there is a delay in subscribing to a new topic for the first time, and then No), a Topic can only correspond to one Producer ID (a Topic belongs to only one Producer, but a Producer can have multiple Topic, the relationship is N:1), and a Topic can correspond to multiple Consumer IDs (a Topic can Belong to multiple consumers, a consumer can subscribe to multiple topics, the relationship is N:N). Topic cannot be used across domains. That is, the Producer ID and Topic must be in the same domain, and the Consumer ID and Topic must be in the same domain.
Common usage of RocketMQ: consistent subscription relationship, cluster consumption and broadcast consumption, message filtering, message retry, consumption idempotence.
The subscription relationship consists of Topic+Tag, the two must be consistent, that is, the subscription relationship is consistent.
The cluster is that the subscribers (instances) of the same Consumer ID belong to the same cluster, and the consumption logic of the subscribers under the same cluster must be exactly the same. The subscriber can logically be regarded as a consumption node.
Cluster consumption mode: MQ believes that any message only needs to be processed by any consumer in the cluster.
Broadcast consumption mode: MQ pushes each message to all registered clients in the cluster to ensure that the message is consumed by each machine at least once. However, no retry operation is performed after the consumption fails.
Consumption details: When starting the Consumer, you can set the number of consumer threads through the ConsumeThreadNums property. If the Consumer ID (consumer) is started for the first time, the messages sent before the startup (ignoring historical messages) will be ignored, and the consumption will start from the messages sent after startup. If it is the second startup, it will start from the last consumption position. Consumption. If you want to start consumption from a specific location, please use the reset consumption location function (only for a specific topic under Consumer ID, does not affect other Consumer IDs).
Message retry: Only valid for cluster consumption mode, broadcast mode does not provide failure retry feature. By default, each message is allowed to retry up to 16 times (customizable). After 16 retries, the message still fails, and the message is discarded. No matter how many times a message is retried, the Message ID of these retried messages does not change. There are three ways to retry: 1. Return Action.ReconsumeLater (recommended); 2. Return Null; 3. Throw an exception.
Consumption idempotency: It is divided into repeated message when sending (Message ID is different, when sending to the server due to network outage or client downtime, the server fails to reply to the client, and the producer realizes that the sending fails and sends again), when delivery The message is repeated (the Message ID is the same, the message has been delivered to the consumer, and the network is interrupted when the client responds to the server. To ensure that the message is consumed once, the server delivers the previously processed message again).
Consumers filter messages according to tags to ensure that consumers only consume the message types they care about.
RocketMQ featured message types: timed and delayed messages, sequential messages, and transactional messages. Several messages are of different message types and are mutually exclusive and cannot be used together (that is, messages cannot be both sequential messages and support timing and transaction messages).
Timing message: A message that is delayed to a certain time after the current time point and delivered to the consumer for consumption. The time point needs to be explicitly specified (must be after the current time point).
Delayed message: (from the current time when the delayed message is sent) a message delivered to the consumer for consumption after a certain delay. The length of delay time needs to be specified. Delayed messages only support the Java language for TCP access.
For timing/delay messages, set a time after the current timestamp through the parameter setStartDeliverTime (must be within 40 days, if the message fails to be sent after 40 days), if this parameter is before the current timestamp, the message will be delivered immediately. If there are messages piled up, timed and delayed messages will be queued behind the piled messages and cannot be delivered strictly according to the configured time. After setting the delivery time of the timed/delayed message, it is still limited by the 3-day message storage time (that is, the message is not consumed after the delivery time point, and the message is deleted after 3 days).
Sequential message: Guaranteed order within the same topic, consisting of sequential publishing and sequential consumption. It is divided into two types: global order and partition order. Sequential messages only support reliable synchronous sending, not asynchronous sending. Sequential messages support cluster consumption, but do not support broadcast consumption. Sequential messages support all MQ public cloud regions and financial cloud regions. For HTTP protocol access, only sequential message sending is supported, and sequential message consumption is not currently supported.
Globally ordered messages: All messages are published and consumed in strict first-in-first-out (FIFO) order (average performance).
Partition sequential messages: All messages are partitioned according to the sharding key (the key field used to distinguish different partitions in sequential messages). Messages within the same partition are published and consumed in strict FIFO order (higher performance).
Transaction message: Implement a distributed transaction function similar to X/Open XA to achieve the final consistency of distributed transactions. The Producer ID of a transaction message cannot be shared with the Producer ID of other types of messages. Semi-message: The message that cannot be delivered temporarily in the transaction message process. The sender has successfully sent the message to the MQ server, but the server has not received the second confirmation of the message from the producer. At this time, the message is marked as "temporary". Undeliverable" state, a message in this state is a semi-message.