How does the message queue thread pool model ensure that messages are not lost when restarting

background

I saw a post on Maimai today, which is more interesting:

The meaning of this post is: When using Kafka, we have set up multiple partitions, how to improve consumption capacity? If you use the thread pool method to improve how to ensure that the message is not lost when restarting.

This question actually asks two points. The first is how to improve the consumption capacity, and the second is how to keep messages from being lost if we choose a thread pool.

Let’s first explain what these two problems are about. In many message queues, there is a concept called partition, which represents partition. Partition is the key to improving the consumption of message queues. The channel for our consumers to consume is from each From each partition, a partition can only be held by one consumer, as shown in the following figure:

It is a bit similar to queuing in a bank. The more the number of queues, the shorter the queuing time will be. Of course, it can also be processed in an asynchronous way, such as a thread pool, where all messages are thrown into the thread pool for execution. This leads to the author's second question. First, let's take a look at why synchronous consumption does not lose messages?

If we are using a synchronous model, we will ack the offset back after we consume it. If we restart without a successful offset, then this part of the data will be consumed again. If it is consumed by a thread pool, how do we proceed? What about ack, for example, we use the thread pool to consume three messages of 10, 11, and 12. If 12 is consumed first, then do we ack 13? If you do this, restart at this time, kafka will think that you have processed the messages of 10 and 11, and the messages will be lost at this time, and the classmates who made this post are more confused about this.

Netizen's answer

Let's take a look at some of the netizens' responses:

Netizen A:

The answer of this netizen is to use the thread pool in essence, and the author also replied, and did not solve the problem of the thread pool.

Netizen B:

This method is similar to queuing in a bank. As long as there are many queues, the processing speed will be accelerated. It is indeed one of the solutions to the first problem.

Netizen C:

This category mainly solves the second problem. By maintaining the offset externally, for example, by entering the offset into the database, we can find the correct offset that should be consumed. This is relatively complicated, and an MQ must be matched with a database. , in case the service I use MQ does not have a database at all, I have to apply for it separately.

Netizen D:

There is another point of view that if the code is written better and the speed of consumption is increased, the consumption capacity will naturally increase. This is indeed a very important point, which is usually ignored by others. Sometimes consumption is slow, and many People may start thinking about how to set up middleware, and often ignore their own code.

After reading a reply to so many posts, I feel that there is no answer that really satisfies me. Here are some ideas in my mind.

my thoughts

For the first question, how to improve consumption power? This problem can be summed up in three ways:

1. If the consumption thread of each consumer machine is fixed, then we can expand the consumption machine and partition, similar to the increase of the queue window in the bank queuing.
2. If the machine and partion are fixed, it is a better way to increase consumption threads, but if it is sequential consumption, the consumption capacity cannot be improved by increasing the number of threads, because each partion in sequential consumption is a separate thread, It can only be solved in the first way.
3. Increase the consumption power of your own code. If you think about the bank's work, if the efficiency of the teller's work can be improved very high, then the entire queuing speed must be very fast.

For the second question, if we use the thread pool model, how to solve the problem of message loss, here I recommend the practice in RocketMQ. We said earlier that it is more complicated to save offsets in the database, and the performance is relatively poor. In RocketMQ The structure of a TreeMap is used in the database to do the things we mentioned above:

private final TreeMap<Long, MessageExt> msgTreeMap = new TreeMap<Long, MessageExt>();

The key of this TreeMap is the offset of each message, and the value is some information of this message. The bottom layer of TreeMap is implemented using red-black trees. We can quickly get the minimum and maximum values. When we process each time When we finish a message, we will remove the message from msgTreeMap,

public long removeMessage(final List<MessageExt> msgs) {
        long result = -1;
        final long now = System.currentTimeMillis();
        try {
            this.lockTreeMap.writeLock().lockInterruptibly();
            this.lastConsumeTimestamp = now;
            try {
                if (!msgTreeMap.isEmpty()) {
                    result = this.queueOffsetMax + 1;
                    int removedCnt = 0;
                    for (MessageExt msg : msgs) {
                        MessageExt prev = msgTreeMap.remove(msg.getQueueOffset());
                        if (prev != null) {
                            removedCnt--;
                            msgSize.addAndGet(0 - msg.getBody().length);
                        }
                    }
                    msgCount.addAndGet(removedCnt);

                    if (!msgTreeMap.isEmpty()) {
                        result = msgTreeMap.firstKey();
                    }
                }
            } finally {
                this.lockTreeMap.writeLock().unlock();
            }
        } catch (Throwable t) {
            log.error("removeMessage exception", t);
        }
        return result;
    }

The removeMessage method is to remove the messages that have been consumed and return the current latest consumption offset. The result returned here is msgTreeMap.firstKey()that the value we ack gives to the message queue server is actually the same. Back to our problem, if we restart, Then there is actually no need to worry that we will lose messages.

finally

Here is just a brief introduction to the message queue to improve the message capability. If you are interested in the message queue, you can read some of my previous articles:

• kafka you must know
• RocketMQ you should know
• In-depth understanding of the use, principle and optimization of RocketMq ordinary messages and sequential messages
• In-depth analysis of how to implement transaction messages

If you think this article is helpful to you, your attention and forwarding are the greatest support for me, O(∩_∩)O:

{{o.name}}

{{m.name}}