Foreword
Github：https://github.com/yihonglei/message-middle
RabbitMQ basis: https://blog.csdn.net/yhl_jxy/category_6979741.html
RocketMQ basis: https://blog.csdn.net/yhl_jxy/category_6979742.html
Kafka basis: https://blog.csdn.net/yhl_jxy/category_6979739.html
Why use a message queue?
Three core scenarios: decoupling, asynchronous, clipping.
1, decoupling
Interface invocation:
If there are A, B, C, D four systems, A system is important data, calls the interface via the downstream B, C, D, respectively, to their different data,
A system is responsible for people with irritable mood tune their interfaces, this time if there are other systems also told him, man, we have data,
We provide an interface written, you've worked hard the next tune, no one let him tune, the person in charge of system A cry, he will have to deal with unusual circumstances of each call timeout,
Brother, you ensure good!
Message decoupled manner:
A system need only to send the corresponding data MQ, B, C, D subscribe to news and consumption do business and deal with it, if we want someone who is responsible for system A transfer interface,
Sipping coffee, tell him with pleasure, subscription XXXTopic, own consumption on the line.
2, asynchronous
Interface invocation:
If there are A, B, C, D system, user A trigger adjusting system, adjusting system A + 30ms + 500ms 200ms = 790ms time B, C, D, user needs, etc. 60ms +,
To see the results after the fast 1s, the data may be processed by the system user A to see the processed data B, C, D of recorded data may be some type of internal our system
Or the user does not need to see the real-time data, these data can be processed asynchronously, the user's patience is limited, do not let the user wait too long, too difficult to obtain the user's heart,
But indeed it is easy to lose.
Asynchronous message:
A core data has been processed, the results returned to the user immediately, and then transmits the corresponding message, B, C, D consume process, the synchronization processing operations into
Asynchronous processing, aims to improve the response speed.
3, clipping
Traffic directly hit the service:
A system if the normal flow is relatively small, but the peak or do activities when traffic is very large, this time the database can not afford so much pressure.
Generally can be solved with a cache or MQ way.
First traffic MQ, A processing system consumption:
Traffic hit to MQ, MQ responsible buffer, A system of batch processing consumption from MQ database according to their own situation.
Second message queue strengths and weaknesses?
Kafka, ActiveMQ, RabbitMQ, RocketMQ compare the advantages and disadvantages?
| characteristic | Availability | Message reliability | Timeliness | Stand-alone Throughput | Support functions | Applications |
| RocketMQ | Distributed architecture to achieve, achieve Netty based communication, very high | After parameter optimization configuration, it can do 0 loss | ms level | 100,000, supporting high throughput | MQ function more perfect, or distributed, scalability | Much business with |
| RabbitMQ | From the main high availability architecture based on high | Basically do not lose, you can open the message persistence function | Microsecond, which is a major feature of RabbitMQ, lowest latency | Ten thousand, lower than RocketMQ, Kafka an order of magnitude | Based on the development of erlang, concurrency is very strong, excellent performance, low latency | Much business with |
| ActiveMQ | 同RabbitMQ | Have a lower probability of data loss | ms level | 同RabbitMQ | MQ extremely functional areas of complete | Much business with |
| Kafka | Distributed, multiple copies of data, very high | With RocketMQ | Within ms delay stage | 100 000, high throughput | Function is relatively simple, the main support simple functions MQ | Real-time computing and large data log collection in the field of large-scale use is |
How to ensure that spending three messages idempotency?
Under normal circumstances, it is to use the MQ message production, consumption message, and to confirm MQ, various different forms of acknowledgment information transmitted from the message queue,
RabbitMQ example, an ACK acknowledgment message is sent, RocketMQ CONSUME_SUCCESS return a success flag, kafka fact, the concept of an offset.
It looks perfect, but there may be duplication of spending in many cases, for example, the reason is because the network flash, ACK returns a failure, and so on failure,
Information is no longer transferred to the message queue, the message queue does not result in the consumer know that they have had the message, and the message is once again available to other consumers.
(Because of the message retry mechanism, etc. If a consumer is broken, rocketmq have consumer cluster, the message will be re-distributed to other consumer).
In general the consumer side processing for repeat consumption, if the consumer end use MQ idempotency did not do, will have a lot of dirty data, businesses may bewildered.
Conventional processing methods such as power consumption:
1, the database level, the establishment of a unique index, insert duplicate exceptionally, no dirty data;
2, Redis of SETNX, distributed lock forms, natural support idempotent;
3, Redis which create a unique id, a record of each consumer, if it already exists, it is no longer consumption;
To deal with the actual idempotency your business, there is necessary an appropriate compensation mechanism needs to be done.
How to ensure that messages are not lost four?
MQ using some non-core scene, it allows lost, but some data is very important, if MQ handle, you need to ensure your message is not lost.
The message may be lost in the production end, are they may not be issued to, may also be sent to the MQ, MQ lost inside, there may not pull messages in the consumer end.
If you are using RocketMQ, he helped us with some processing messages are not lost.
RocketMQ substantially message flow process
producer (producers to produce news) - "broker (stored messages) -" cunmser (consuming messages)
How producer to ensure that messages are not lost:
1, by default, by way of blocking the transmission of synchronization, check SendStatus, the status is OK, the message indicating successful delivery to certain Broker,
Timeout or failure, the default 2 retries is triggered. Send the results of this method, the success of the Broker may store, or may not succeed.
2, the way to take delivery of transaction messages, the message does not guarantee 100% success delivered to the Broker, but if the message is sent Ack fails,
This message is stored in CommitLog them, but ConsumerQueue is not visible. This anomaly can view the message in the log,
Strictly speaking, it is not completely lost.
3, RocketMQ support indexing logs, after a timeout if a message is sent, it can also query logs API, to check whether the success of the Broker storage.
How broker to ensure that messages are not lost:
1, the message persistence to support Commitlog inside, even down after the restart, the message is not consumed can be loaded out.
2, Broker brush disc itself supports synchronous, asynchronous brush plate strategy can ensure a certain received message stored in local memory.
3, Broker cluster supports N from the main strategy to support the way synchronous replication and asynchronous replication, synchronous replication ensures that even if the Master disk crashes, the message is still not lost.
How cunmser to ensure that messages are not lost:
1, Consumer itself maintains a persistent offset (corresponding to the MessageQueue inside min offset), marking the success of consumption or have already successfully sent back to the broker
Subscript messages.
2, if the Consumer consumption fails, then it will put this message back to the Broker, the back is successful, then update their offset.
3. If the Consumer consumption fails when sent back to the broker, broker hung up, then the Consumer periodically retry this operation.
4, if the broker and Consumer linked together, the message is not lost, because the consumer is a timing offset which is persistent, after the restart,
Continuing with the previous message to the local offset pull.
How to ensure message ordering five consumer?
Sometimes the message we send is the order issued by the consumer, we also need to order the consumer, if the disorder will lead to data anomalies.
Each MQ common approach is to build multiple queue, when sending the order to ensure consumer needs are the order of the messages sent to the same queue,
End consumer need in FIFO order queue consumer, if the consumer uses multi-threading, multiple worker to deal with, we need to pay attention,
Multithreading consumer you can not guarantee the order in each thread processing, it requires special care.
Continuously updated perfected ......
