Cause Message disorder arising
Message queues, since it is able to ensure that the message queue to ensure orderliness message when entering the queue, and the queue, it is clearly in the production end of the message (Producer), but often there are multiple messages in a production environment spending end (consumer), although the consumer pulling the end of the message is ordered, but because of various aspects of message networks can not guarantee the consumer side of the respective processing order.
Scene analysis
Twice the modified product information, messages A and B have synchronized write message MySQL, then send message queue asynchronous write message, then the message queue production side (Producer) chronologically successively sent two messages A and B ( A first message is sent, the sent message B). By business logic, the final state of the product information needs to be integrated messages A and B prevail message.
A seemingly more common synchronous write to the database, asynchronous scene to send the message, but in fact the need to ensure an orderly consuming messages.
- Hypothesis 1: A message containing only the modified trade name , message B contains only the modified product weight at this time the consumer end of the message queue in fact need not be concerned message sequence , message queue consumer side (Consumer) just to consumption.
- Assumption 2: A message containing the modified trade name, weight , message B containing the modified trade name , when the consumer receives the first end of the message B, the message A is received, then the modified message B will be overwritten. At this time the consumer end of the message queue and in fact need to focus on the message timing .
Visible, you can not guarantee what the message contains information, must now ensure the orderly consumption message.
How the business point of view to ensure orderly news consumption
- Production end when sending a message, the message is always to ensure that the full amount of information.
- Consumer terminal when receiving messages by way of the cache timestamp, is determined whether the latest time when the message generating consumption message is discarded if it is not, and if so the next step.
The following pseudo code is described by way of:
Production end pseudocode
insertWare (ware); # insert data into the database, usually when inserted into the database we will only update modified fields, but not the full amount of insertion
ware = selectWareById (ware.getId); # Get the total amount of product information (at this time is the latest), to put it into the message queue
syncMq (ware); # A message sent asynchronously mq
Consumer end pseudo code
ware = fetchWare (); # Get message
if (isLasted (ware)) # commodity by modifying the timestamps to determine whether the latest changes
TODO # business logic to the next step
else
discards the message return #
Focuses on how the consumer side to determine whether the message is the latest modification is the isLastedmethod.
isLasted way
Long modified = getCacheById (ware.getId); # Get the latest modification time of the article commodity cache
If (ware.getModified> modified) {# Product Review If the message is longer than the time register, indicating the latest operation
setCacheById (ware); # piece of merchandise modification timestamp messages written to the cache
to true return;
} # the else if the product modified message is less than the time of the cache, the article described the message belongs to the "operation history" does not update it return false;
The above is by way of pseudo-code that describes how through business means to ensure orderly news consumption, with emphasis on the total amount of caches and send a message timestamps . Where there are some technical implementation details.
For example: the consumer side consumption message B, to perform after acquiring the time stamp buffer, and re-set before the new cache , just when the other end consumer consumption is also being performed to B it acquires a time stamp buffer , since this message A when not update the cache, the cache still get the message a cache is old, then you will agree that there are two consumer end when his own consumption latest news, causing the dropped messages Modiu.
Obviously, this is a distributed thread safety issues, distributed lock typically use Redis or ZooKeeper, execution timing of the lock as shown below.
This is the message from a business point of view to ensure the orderly consumption at the consumer end. Send messages sent in the message of the full amount and the time stamp in the message the consumer side caching can ensure the orderly consuming messages.
In the above scenario is to write synchronization MySQL, then the total amount of data acquired product, followed by asynchronous message transmission. This series of steps can be achieved by bonding the MySQL binlog, after synchronous write MySQL, MySQL binlog change transmission, received by MySQL binlog change Alibaba Canal middleware then sends a message to the message queue.
