This article from the yanglbme starting on GitHub technology community Doocs , currently stars exceeded 30k.
Project Address: github.com/doocs/advan...
Interview questions
How to guarantee the order of the message?
Interviewer psychological analysis
This is the time to ask the message queue will ask the topic, see if you do not know the first order of this thing? Second look at you there is no way to ensure that the message is in order? This is a common problem in the production system.
Face questions analysis
I, for example, we have done before a mysql binlogsynchronization system, the pressure is still very large, daily to reach hundreds of millions of data synchronization, data that is unchanged from a synchronization mysql mysql library to another library to go inside (mysql -> mysql). It said common point is that such a large data team, you need to synchronize a mysql database over the company's business system data to do complex operations.
You add or delete a change in mysql in data, the corresponding out additions and deletions to 3 binloglog, then the three binlogsent to the MQ inside, then out of order execution consumption, at least you have to ensure that people are in the order, huh? Otherwise it would have been: to add, modify, delete; you are shocked to change the order of execution to delete, modify, add, incomplete wrong yet.
Actually, this data synchronization over, this should be the last data has been deleted; the results you made a mistake in that order, and finally the data preserved, data synchronization on the wrong.
Take a look at the order will be garbled two scenarios:
- RabbitMQ : a queue, more consumer. For example, where the producer sends three data to RabbitMQ, the order is data1 / data2 / data3, RabbitMQ is pushed to a memory queue. There are three consumer a consumer of these three data from MQ, the results of the consumer 2 to performing the operation, the data2 stored in the database, then data1 / data3. This is not obvious chaos.
- Kafka used to live : for example, we built a topic, there are three partition. Producers in writing, in fact, you can specify a key, for example, we specify an order id as the key, then the order-related data, will be distributed to the same partition go, and the data in this partition must It is in order.
When consumers take out data from the partition, it must be sequential. Here, the order is ok, there is no confusion. Then, we might engage consumers in multiple threads to concurrently process messages . If the consumer because the consumer is a single-threaded process, while time-consuming process, then such time-consuming process a message several ms, then one second can handle dozens of messages, this throughput is too low. While concurrently run multiple threads, it might mess up the order.
solution
RabbitMQ
A plurality of split queue, each queue a consumer, it is a little more queue, is troublesome indeed point; or it corresponds to a queue, but a consumer, then the consumer do queuing with internal memory queue, then distributed to the different underlying worker deal with.
Kafka
- A topic, a partition, a consumer, internal consumption single-threaded, single-threaded throughput is too low, generally do not use this.
- The N memory write queue, the same key data to the same memory queue; then for N threads, each respective memory queue to a consumer in order to assure sequential.
I welcome attention to the micro-channel public number "Doocs the open source community," the first time to push original technology articles.