What is messaging middleware?
Semantically accurate messages are passed between different systems by providing some kind of specification.
An integrated distributed system that focuses on sending and receiving data and utilizes an efficient and reliable asynchronous messaging mechanism.
What is MQ?
The full name of MQ is Message Queue. Message Queuing (MQ) is a communication method for applications "to" applications, and it is also a kind of message middleware.
MQ: Producers write messages to the message queue, and consumers can read messages in the queue.
Application Scenarios of Message Queuing
a. Asynchronous processing: such as the callback notification that the order status is processed;
b. Application decoupling between systems: the former system puts the content to be processed into the message queue, and no longer cares about other subsequent operations, and the latter system obtains the message for consumption;
c. Traffic shaving: avoid excessive traffic, resulting in a surge in traffic, accumulation of massive messages, and application hangup;
d. Log processing: log processing through the queue;
e. Message communication: Message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication.
basic concept
1 message model
Peer-to-peer model (P2P):
In the P2P model, there are the following concepts: message queue (Queue), sender (Sender), receiver (Receiver). Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues hold messages until they are consumed or timed out.
Each message has only one consumer (Consumer) (that is, once consumed, the message is no longer in the message queue).
There is no time dependency between the sender and the receiver, which means that when the sender sends a message, it does not affect the message being sent to the queue regardless of whether the receiver is running or not .
· The receiver needs to acknowledge success to the queue after successfully receiving the message.
If you want every message sent should be successfully processed, then you need a P2P model.
Publish-subscribe model (Pub/Sub):
In the Pub/Sub model, there are the following concepts: topic (Topic), publisher (Publisher), subscriber (Subscriber). Clients send messages to topics. Multiple publishers send messages to a topic, and the system delivers those messages to multiple subscribers.
Each message can have multiple consumers
· There is a time dependency between publishers and subscribers. For a topic (Topic) subscriber, it must create a subscription before it can consume the publisher's message, and, in order to consume the message, the subscriber must keep running.
Subscribers create a durable subscription. This way, even if the Subscriber is not activated (running), it can receive the Publisher's messages.
If the message you want to send can be processed without any processing, or can be processed by one consumer, or can be processed by multiple consumers, then the Pub/Sub model can be used.
2 Interaction Principles
What do nameservers do?
Namesrv is mainly used as a routing service (state server), and it is stateless and can be clustered horizontally. The broker simultaneously reports the real-time status information of the broker machine, topic message queue routing information, broker basic information, and cluster information to multiple Namesrv machines, so as to achieve the function of data hot backup. When the Producer sends a message, it first requests nanesrv to obtain the topic routing information, and then routes to the broker according to the topic routing information (the topic is stored in the broker). When Consumer consumes messages, it first requests nanesrv to obtain topic routing information, and then routes to the broker according to the topic routing information.
What do brokers do?
Message transfer role, responsible for storing and forwarding messages. Broker deployment is relatively complicated, divided into master and slave. One master can correspond to multiple slaves, but one slave can only correspond to one master (supports asynchronous replication and semi-synchronous replication). The master can be deployed in a cluster, which is distinguished by the same ClusterName, and the BrokerName in the same cluster cannot be the same. Each broker establishes a long connection with all nodes in the namesrv cluster, sends heartbeat information (30 seconds) regularly, and registers information such as topics to all namesrv.
Producer
The Producer establishes a long connection with one of the nodes in the Name Server cluster (selected at random), regularly fetches Topic routing information from the Name Server, establishes a long connection to the Master that provides Topic services, and sends heartbeats to the Master regularly, and the Slave will synchronize the Master's information. Producer is completely stateless and can be deployed in clusters.
Consumer
The Consumer establishes a long connection with one of the nodes in the Name Server cluster (selected at random), periodically fetches Topic routing information from the Name Server, establishes a long connection to the Master and Slave that provide Topic services, and sends heartbeats to the Master and Slave regularly. Consumers can subscribe to messages from the Master or from the Slave, and the subscription rules are determined by the Broker configuration.
Broker cluster
Multiple Slaves can be mounted under one Master, and multiple Slaves under the same Master can be distinguished by specifying different BrokerIds.
3 Test content and troubleshooting methods
Q1: Can MQ guarantee that messages are not repeated?
1. In most cases, messages are not repeated. As a distributed message middleware, in abnormal situations such as network jitter and application processing timeout, it cannot guarantee that messages will not be repeated, but it can guarantee that messages will not be lost;
2. The business side needs to implement deduplication logic by itself. It is not recommended to use msgid for deduplication, because in most cases, msgid will not be repeated, and it is not ruled out that duplication may occur in extreme cases, or between multiple clusters. repeat msgid;
3. You can use the unique identifier such as the order number to deduplicate.
Q2: MQ has not been consumed, and the message has been sent, but how to query if it has not been received?
MQ provides a variety of message query methods:
1. Use Topic to query by time range, you can query all messages received by a Topic within a certain period of time;
2. Use Topic and MessageID to accurately query messages;
3. Use Topic and MessageKey to more accurately query a class of messages with the same MessageKey.
Q3: How to re-consume messages if MQ consumption fails?
1. If the consumption business logic code returns ConsumeConcurrentlyStatus.RECONSUME_LATER, or NULL, or throws an exception, the message will go through the retry process, and the default retry is 16 times (the number of retries can be customized). If it still fails after 16 retries , the message is discarded;
2. The message retry interval can be controlled by context, if not controlled, it will become longer and longer.
Q4: How to troubleshoot MQ consumption failure and container environment message consumption failure?
1. Check whether the subscription relationship is correct and whether the topic is correct;
2. Check whether the status of the consumer group is normal;
sh[username]consumerConnection-n127.0.0.1:9876-g consumer group name
sh[username]consumerProgress-n127.0.0.1:9876-g consumer group name
3. Check whether the address resolution of the producer machine and the consumer machine is the mq address of the container group; configure and view it in the template configuration of the docker management platform;
4. Check whether the consumer server is configured with an external network address, causing the message to be sent to the benchmark environment;
5. Confirm that the producer has produced the message.
Q5: Simple log query
Query the records of put and pull through the log on the MQ server.
Q6: MQ consumption precedes synchronous return
Generally, a locking mechanism is used for processing.
4 naming convention
Topic naming convention:
1. All topics must start with T_. Topics contain business systems, which can uniquely distinguish businesses.
2. The tag must be filled in. The same topic is distinguished by different tags, all capitalized. Consumers consume messages through Topic+tag.
3. Use the operation and maintenance platform to check whether the topic exists. If it exists, you need to consider whether to replace the topic.
例：topic：T_LOAN_PRODUCT
tag: T_CANCEL_BORROW or T_SUBMIT_RESALE[topic+tag matching consumption message]
Consumer group naming convention:
1. The consumption group must start with S_, including the business system, all capitalized. The same topic can be consumed by multiple consumer groups.
2. Use the operation and maintenance platform to check whether the consumption group exists. If there is, you need to consider whether to replace the consumption group.
Example: c-group-id:S_PFD_T_ATM
Production group naming convention:
The producer must start with P_, including the business system, all capitalized, different systems are prohibited from using the same group name.
Example: p-group-id:P_PAYFUND
Finally: The complete software testing video tutorial below has been sorted out and uploaded, and friends who need it can get it by themselves [Guaranteed 100% free]
Software Testing Interview Documentation
We must study to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Ali, Tencent, and Byte, and some Byte bosses have given authoritative answers. Finish this set The interview materials believe that everyone can find a satisfactory job.
