RabbitMQ
1. RabbitMQ-How to ensure that messages are not lost?
At that time, we used RabbitMQ to achieve double-write consistency of data between MYSQL and Redis. This required high availability of messages, and we had to ensure that messages were not lost.
Mainly consider from three levels
- The first is to enable the producer confirmation mechanism to ensure that the producer's messages can reach the queue. If an error is reported, it can be recorded in the log first and then the data can be repaired.
- The second is to enable the persistence function to ensure that messages will not be lost in the queue before being consumed. The switches, queues, and messages must be persisted.
- The third one is to turn on the consumer confirmation mechanism to auto. After spring confirms that the message is processed successfully, ACK is completed. Of course, a certain number of retries also needs to be set. We set it 3 times at the time. If the message is not received after 3 retries, The failed message is delivered to the abnormal switch for manual processing.
2. How to solve the problem of repeated consumption of RabbitMQ messages?
We have actually encountered this, and it is like this. At that time, our consumers set up an automatic confirmation mechanism. Before the service had time to confirm it to MQ, the service went down, causing the service to restart and consume the message again. This results in repeated consumption.
Because the payment we were processing (order|business unique identifier) had a unique business identifier, when we processed the message, we first went to the database to check whether the data existed. If it did not exist, it meant that it had not been processed. At this time This message can be processed normally. If this data already exists, it means that the message has been consumed repeatedly, and we don't need to consume it anymore.
2.1. Do you know of other solutions?
In fact, this is a typical idempotent problem. For example, redis distributed locks and database locks are both possible.
3. What is the dead letter switch in RabbitMQ? (Have you ever understood the RabbitMQ delay queue?)
Our xx project at that time had a xx business that required the use of delay queues, which was implemented using RabbitMQ.
The delay queue is implemented using a dead letter switch and TTL (message survival time).
If a message times out and is not consumed, it will become a dead letter. In RabbitMQ, if a message becomes a dead letter, the queue can be bound to a dead letter switch. On the dead letter switch, other queues can be bound. When we send messages, we can bind them as needed. Specify the time of TTL, so that the function of delay queue is realized.
I remember that RabbitMQ has another way to implement a delay queue. It is more convenient to install a dead letter plug-in in RabbitMQ. We only need to specify this as the dead letter switch when declaring the interactive machine, and then directly send the message Just specify the timeout period. Compared with dead letter switch + TTL, some steps are omitted.
4. If there are 1 million messages piled up in MQ, how to solve it?
I have never encountered this situation in actual development. However, if there are accumulated problems, there are many solutions.
- First: To improve consumers’ consumption power, you can use multi-threaded consumption tasks
- Second: Add more consumers to increase consumption speed.
Use the work queue mode to set up multiple consumers to consume messages in the same queue. - Third: Expand the queue volume and increase the accumulation limit
You can use RabbitMQ lazy queue. The main benefits of lazy queue are
- ① After receiving the message, it is stored directly in the disk instead of the memory.
- ② When consumers want to consume messages, they read them from the disk and load them into memory.
- ③Support millions of message storage.
5. Have you ever understood the high availability mechanism of RabbitMQ?
Our project was in a production environment at that time, and the cluster we used was a mirror mode cluster, using 3 machines.
The mirror queue structure is one master and multiple slaves. All operations are completed by the master node and then synchronized to the mirror node. If the master node goes down, the mirror node will be replaced by the new master node. However, before the master-slave synchronization is completed, the master node It is down and data may be lost.
5.1. How to solve the problem of data loss?
We can use arbitration queues, which, like mirror queues, are in master-slave mode and support master-slave data synchronization. Master-slave synchronization is based on the Raft protocol and is strongly consistent.
And it is very simple to use. No additional configuration is required. When declaring the queue, you only need to specify that this is the arbitration queue.
kafka
1. How does Kafka ensure that messages are not lost?
There are many guarantee mechanisms. From sending messages to consumers receiving messages, messages may be lost at each stage, so when we solve it, we also consider it from multiple aspects.
- The first is that when the producer sends a message, it can use an asynchronous callback to send. If the message fails to be sent, we can obtain the message information after the failure through the callback. We can consider retrying or logging, and we can make compensation later. At the same time, message retry can also be set on the producer side. Sometimes the transmission is unsuccessful due to network jitter, which can be solved by using the retry mechanism.
- Second, messages may be lost in the broker. We can use Kafka's replication mechanism to ensure that messages are not lost. When the producer sends a message, you can set an acks, which is the confirmation mechanism. We can set the parameter to all. In this case, when the producer sends the message to the partition, the confirmation will not only be saved in the leader partition, but also in the follower partition. Only when all replicas save the confirmation will it be considered successful. Message, so this setting can largely guarantee that the message will not be lost in the broker.
- The third possibility is that the message is lost on the consumer side. Kafka consumption messages are marked and consumed according to the offset. The consumer defaults to automatically submitting the consumed offset on schedule. The default is to submit it every 5 seconds. If there is a repeat In the case of balance, repeated consumption or data loss may occur. We generally disable the automatic price increase offset and instead submit it manually. When the consumption is successful, we will report the consumption position to the broker, so as to avoid message loss and repeated consumption.
2. How to solve the problem of repeated consumption of messages in Kafka?
Kafka consumption messages are marked and consumed according to the offset. The consumer defaults to automatically submitting the consumed offset on schedule. The default is to submit every 5 seconds. If a rebalancing occurs, repeated consumption or data loss may occur. We generally disable the automatic price increase offset and instead submit it manually. When the consumption is successful, we will report the consumption position to the broker, so as to avoid message loss and repeated consumption.
In order to make messages idempotent, we can also set a unique primary key to distinguish, or lock, database lock, or redis distributed lock, which can solve the idempotent problem.
3. How does Kafka ensure the order of consumption?
Kafka's default storage and consumption of messages cannot guarantee sequentiality, because a topic data may be stored in different partitions, and each partition has an offset stored in order. If the consumer is associated with multiple partitions, it cannot Ensure orderliness.
If there is such a demand, we can solve it. Just store the messages in the same partition. There are two ways to set it. The first is to specify the partition number when sending the message, and the second is to send the message. When setting the same key according to the same business, because by default the partition is also selected by the hashcode value of the key. If the hash value is the same, the partition must be the same.
4. Have you ever understood the high availability mechanism of Kafka?
There are mainly two levels, the first is the cluster, and the second is the replication mechanism.
A Kafka cluster is composed of multiple broker instances. Even if one of them goes down, it will not delay other brokers from continuing to provide services to the outside world.
The replication mechanism can ensure the high availability of Kafka. A topic has multiple partitions, and each partition has multiple copies. There is a leader, and the rest are followers. The copies are stored in different brokers; the contents of all partition copies are They are all the same. If the leader fails, one of the followers will be automatically promoted to leader, ensuring the fault tolerance and high availability of the system.
4.1. Explain the ISR in the replication mechanism
ISR means in-sync replica, which is a follower that needs to be replicated and saved synchronously.
There are many followers in the partition copy, which are divided into two categories. One is ISR, which saves data synchronously with the leader copy. The other is an ordinary copy, which synchronizes data asynchronously. When the leader dies, one will be selected from the ISR copy list first. As a leader, because the ISR saves data synchronously and the data is more complete, the ISR copy list is preferred.
5. Do you understand the Kafka data cleaning mechanism?
Topic data in Kafka is stored in partitions. If the file in the partition is too large, segments will be stored in segments.
Each segment is stored on the disk in the form of an index (xxxx.index) and a log file (xxxx.log). The advantages of such segmentation are that, first, it can reduce the size of a single file content and make it easier to find data. Second, Convenient for Kafka to clean logs.
Kafka provides two log cleaning strategies:
- First, according to the retention time of the message, when the message storage time exceeds the specified time, cleanup will be triggered. The default is 168 hours (7 days).
- The second is based on the size of the data stored in the topic. When the size of the log file occupied by the topic is greater than a certain threshold, the oldest message will be deleted. This is turned off by default.
Both strategies can be set through the configuration file in Kafka's broker.
6. Have you ever understood the design to achieve high performance in Kafka?
Kafka's high performance is the result of collaboration in many aspects, including macro architecture, distributed storage, ISR data synchronization, efficient use of disks, operating system features, etc.
The main points are as follows:
- Message partitioning: It is not limited by a single server and can process more data without limitation.
- Sequential reading and writing: Sequential reading and writing of disks improves reading and writing efficiency.
- Page cache: Cache the data in the disk into the memory, changing the access to the disk into the access to the memory.
- Zero copy: Reduce context switching and data copying.
- Message compression: Reduce disk IO and network IO.
- Send in batches: Pack messages and send them in batches to reduce network overhead.
Finish! ! !
明天对于世界而言，永远是一个奇迹。---柏拉图