Article directory
1. How to prevent repeated consumption of messages
In the solution to prevent message loss, if the producer has not received the ACK returned by Kafka to confirm receipt of the message due to network fluctuations after sending the message, but in fact the Broker has received the message, the producer will restart at this time. Send message data and retry, so Broker will receive multiple pieces of the same message data, resulting in repeated consumption by consumers.
How to solve the problem that the producer sends repeated messages?
- The producer turns off the retry mechanism, but it will cause data loss, so this method is not recommended.
- Solve the problem of non-idempotent consumption through consumers:
- The so-called idempotence means that the results of multiple visits are the same. For rest requests (get (idempotent), post (non-idempotent), put (idempotent), delete (idempotent))
- A joint primary key can be created in the database to prevent the same primary key from creating multiple records.
- Use distributed locks and use the business id as the lock to ensure that only one record is created.
2. How to ensure the sequential consumption of messages
The so-called sequential consumption refers to: messages are consumed one by one. For example, in a business scenario, there are three messages. After the first message is consumed, an order is placed, after the second message is consumed, it is payment, and after the third message is consumed, it is shipped. , it must be ensured that messages are consumed sequentially, otherwise data errors will occur.
The method to ensure the sequential consumption of messages:
- Producer: Ensure that the messages are consumed in order, and the number of messages will not be lost, use synchronous sending, and set ack to a value other than 0.
- Consumer: Only one partition can be set for the topic, and there can only be one consumer in the consumer group.
There are not many usage scenarios for Kafka's sequential consumption, sacrificing performance.
3. How to solve the problem of message backlog
The consumer's consumption speed is far behind the producer's production speed, which will lead to a large amount of unconsumed data in Kafka. As more and more unconsumed data accumulates, it will lead to The performance of consumer addressing is getting worse and worse, and finally the performance of providing services to the entire Kafka is very poor.
Solutions for message backlog:
- In the consumer, use the multi-thread mode to make full use of the performance of the machine for consumption.
- Improve the performance of business-level consumption through business architecture design.
- Create multiple consumption groups and multiple consumers, deploy them in different servers, and consume at the same time.
- Create a consumer. The consumer creates a topic in Kafka, configures multiple partitions, and configures multiple consumers for each partition. The consumer forwards the polled messages directly to the newly created topic without consumption. At this time , multiple consumers of multiple partitions of the new topic start to consume together, which is not commonly used.
4. Implementation of delay queue
After the order is created, if there is no payment for more than 30 minutes, the order needs to be canceled. How to judge the 30-minute interval between the time of placing the order and the current time can be realized through the delay queue.
Implementation of the delay queue:
- A topic is created in Kafka as a delay queue, and consumers consume messages in this topic.
- When a consumer consumes a message, it is judged whether the creation time of the message and the current time exceed 30 minutes, provided that the order has not been paid.
- If it has been paid, then go to the database to modify the order status to paid.
- If there is no payment, then record the message data whose offset value of the current message will not be after consumption, wait for 1 minute, and then pull the message after the offset from Kafka again, and continue to judge.
Not quite sure what the red part is about.