Message Queue interview topics

How to avoid the duplication of spending

RocketMQ is no guarantee that the message does not repeat, if your business does not need to be rigorously duplicate messages, you need to go heavy on the business end

Interface idempotency protection, consumer-side processing business news to keep idempotency

Redis

1) setNX (), the message id to do heavy, java version currently does not support setting expiration time

// the Redis operation, has been operated is determined whether the TODO 
Boolean In Flag = jedis.setNX (Key);
 IF (In Flag) {
     // consumption 
} the else {
     // Ignore, repeated consumption 
}

Expanded (if not the atomic expire again, the following manner may be distributed lock)

Lock
String result = jedis.set(key, value, "NX", "PX", expireTime)

Unlock (Lua script, first check the key, and then release the lock match, lua can guarantee atomicity)
String script = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end";
Object result = jedis.eval(script, Collections.singletonList(lockKey), Collections.singletonList(requestId));

Note: lockKey can be a commodity id, requestId used to identify the same clients

2) Incr atomic operation: key increment, the return value is greater than 0 then the consumer through

int NUM = jedis.incr (Key);
 IF (NUM ==. 1 ) {
     / Consumption
} The else {
     // ignore, repeated consumption 
}

Both methods can be, but not in a distributed lock problems considered atoms, but may not be considered atoms duplication problems, large amount of data needs to set an expiration time.

De-duplication database table

Message in a field using a unique index key do

 

How to ensure the reliability of the message, the message is lost in processing problems

producer 端

• Do not use oneway send, send synchronous or asynchronous mode, do the retry, retry, but the Message key must be unique
• Delivery logs need to be saved, the key field, delivery time, delivery status, number of retries, the request and to respond to the body

broker end

• Double double master-slave architecture, NameServer multiple nodes are required
• Simultaneous dual writing, asynchronous brush disc (synchronous brush disc is more reliable, but the performance is almost, according to the service selection)

consumer 端

• Consumer sure to keep a log message, that message metadata and message body
• Be sure to check the consumer side idempotency treatment

After delivery to the broker terminal

• Machine Cycle power: asynchronous brush plate, the message is lost; synchronization message is not lost brush disc
• Hardware failure: there may be lost, see queue architecture

 

If the message is a lot of accumulation in the broker inside, how they deal

Line failure, and how to deal with

• News piled up 10 hours, tens of millions of messages to be processed, and now how to do?
• Repair consumer, then slowly consume? It can also take several hours to complete consumption, the new message is how to do?

Correct posture

• Topic temporary queue expansion, and improve the ability of consumers, but if you increase the number of Consumer, but the accumulation of a fixed number of topic inside the message queue, too many consumer can not be assigned to a message queue
• The preparation of interim manipulating dispatcher, quickly read from the old to the new interim Topic Topic in New Topic expansion of the number of times the queue, and then start more consumer in temporary consumption in new Topic

 

RocketMQ cause analysis of high-performance, high-availability architecture

MQ architecture Configuration

• Sequential write, random read, zero-copy
• Synchronous and asynchronous SYNC_FLUSH brush disc brush plate ASYNC_FLUSH, arranged by flushDiskType
• Synchronous and asynchronous replication, by brokerRole configuration ASYNC_MASTER, SYNC_MASTER, SLAVE
• Recommended synchronous replication (double write), asynchronous disk brush

Availability transmitting end

• Dual dual master-slave architecture: Create Topic corresponding time, MessageQueue created on multiple Broker
• Broker i.e., the same name but different brokerid (i.e., master-slave mode) when a Master is unavailable, the other group Master still available.
• But when sufficient machine resources, the need to manually turn into a slave master, currently does not support automatic conversion, usable shell processing

Consumer availability

• Master-slave architecture: Broker role, Master provides read and write, Slave read-only support
• Consumer no configuration, or when busy with the Master, Slave Consumer will automatically switch to the node can be read

Increase the spending power of the message

• Parallel consumption
  • Add multiple nodes
  • Consumer individual increase parallelism, modifications and consumerThreadMax consumerThreadMin
  • Volume consumer, set consumerMessageBatchMaxSize Consumer, the default is 1, if N, message and more, each time the message is received the N
• Select Linux Ext4 file system, Ext4 file system size 1G deleted files usually takes less than 50ms, and Ext3 file systems require time-consuming 1s, when you delete files from the disk IO great pressure, can cause IO operation timed out