Apache pulsar technology series--several ways of message retweeting

Introduction

Apache Pulsar is a multi-tenant, high-performance inter-service message transmission solution that supports multi-tenancy, low latency, read-write separation, cross-regional replication (GEO replication), rapid expansion, and flexible fault tolerance. In many scenarios, users need to implement message re-push capabilities through MQ, such as timeout re-push, re-push when handling exceptions, etc. This article introduces several message re-push solutions provided by Apache Pulsar.

In the actual use of MQ, when users consume data, they may encounter message processing exceptions or scenarios where processing needs to be postponed. This involves message retweeting logic. Pulsar itself provides the ability to retweet messages. This article mainly introduces the message retweet mechanism of Pulsar.

Message acquisition (pull/push) mechanism

The consumption of Pulsar adopts a combination of push and pull message acquisition mechanism. Before the Consumer obtains the message, it will first notify the Broker (FLOW request), and the Broker will push the message to the Consumer according to the configured ReceiveQueue size and the number of messages that the Consumer can currently receive.

The detailed interaction process is shown in the figure below:

After the Consumer is created, it will use the size of MaxReceiveQueue as the Permit value, which is the maximum number of messages that the Consumer can cache.
Then, Consumer initiates a FLOW request to Broker, carrying Permit information (Consumer Permit is reduced to 0), Broker will record this Permit as Consumer's AvailablePermit after receiving it, and AvailablePermit determines the amount of data that Broker can send to Consumer (actually when reading data judge).
If AvailablePermit > 0, Broker starts to read data (assuming there are N items), and then pushes it to Consumer. After pushing, AvailablePermit is decremented by N.
After the Consumer receives the message, it will not return it directly to the user, but put it in the ReceiveQueue. When the user calls the Receive() method to get the message, the Consumer will Permit + 1.
When Permit > MaxReceiveQueueSize / 2, Consumer will initiate a Flow request again with the current Permit value.

The above process is the message passing process between Consumer and Broker.

By default, after the data is pushed to the Consumer, it is completely handed over to the user for processing, and the data will not be pushed repeatedly. This method cannot meet the scenarios that require retweeting. The following introduces several retweeting mechanisms of Pulsar.

SDK Unification re-launch

A more intuitive approach is to re-push the message if there is no Ack after a certain period of time.

Currently, Pulsar provides the ability to control data re-push through the timeout period. Consumers can configure AckTimeout (off by default). After setting AckTimeout, Client will build an UnAckedMessageTracker, and all messages of the user Receive() will be tracked by UnAckedMessageTracker. When the user Acks the message, it will be deleted from UnAckedMessageTracker. For messages without Ack, UnAckedMessageTracker will have a scheduled task to check. If the AckTimeout time has passed, it will trigger a retweet.

Re-push is realized through RedeliverUnackMessage, UnAckedMessageTracker will actively initiate a Redeliver request, and Broker will re-push according to the requested MessageId information.

AckTimeout is set when Consumer is initialized:

Consumer<String> consumer = pulsarClient.newConsumer(Schema.STRING)
                                                              .ackTimeout(10, TimeUnit.SECOND)

Re-introduction of user decision--NegativeAck

The retweeting through AckTimeout is implemented uniformly within the SDK, and the user cannot control the retweeting behavior. If the user wants to decide which messages need to be retweeted according to his usage scenario, Pulsar provides the NegativeAck capability.

NegativeAck is similar to AckTimeout. There is a NegativeAcksTracker to manage message retweeting. NegativeAcksTracker will only track the MessageID of the user actively calling the NegativeAcknowledge() method. The retweeting logic is also implemented through RedeliverUnackMessage.

NegativeAck can set the Delay time of Redelivery.

 Consumer<String> consumer = pulsarClient.newConsumer(Schema.STRING)
                              .negativeAckRedeliveryDelay(1001, TimeUnit.MILLISECONDS)

When used, it needs to be called explicitly.

// call the API to send negative 
acknowledgmentconsumer.negativeAcknowledge(message);

Re-introduction of User Decision--RLQ

In addition to the NegativeAck method, users can also implement active message retweeting through the retry queue (RLQ). RLQ is generally used in scenarios where the user cannot process some messages temporarily and wants to process them later.

Pulsar provides the ReconsumeLater() method to implement the retry queue. Unlike Negative, RLQ will create a new Topic. The format of the Topic is TopicName-SubscriptionName_RLQ. Every time ReconsumeLater(), a new message will be written. to the RLQ Topic, and will Ack the previous message.

After setting the Consumer of RLQ, the SDK will start the subscription of RLQ by default, so the news of RLQ will also be consumed by the Consumer.

RLQ is configured via DeadLetterPolicy (DLQ is explained below).

Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
 	  .topic("my-topic")
 	  .subscriptionName("my-subscription")
	  .subscriptionType(SubscriptionType.Shared)
 	  .enableRetry(true)
	  .deadLetterPolicy(DeadLetterPolicy.builder()
 	  .maxRedeliverCount(maxRedeliveryCount)
	  .build())
     .subscribe();

The following information will be added to the message properties in the RLQ Topic:

Special property	Description
REAL_TOPIC	Original Topic name
ORIGIN_MESSAGE_ID	Original MessageId
RECONSUMETIMES	The number of repeated consumption
DELAY_TIME	delivery delay

RLQ also needs to actively call: consumer.reconsumeLater(msg, 3, TimeUnit.SECONDS).

Add a limit to the number of retweets --DLQ

For continuous data processing failures, it is not a good strategy to keep retrying. At this time, Dead Letter Queue (DLQ) is a better choice. DLQ allows users to write data that continues to fail to be processed to an independent Dead Letter Topic , DLQ data requires a separate subscription to consume.

The format of DLQ Topic is TopicName-SubscriptionName_DLQ. DLQ needs to set an upper limit for retries. When the number of retries exceeds the upper limit, it will be written into the DLQ Topic.

Consumer<byte[]> consumer = pulsarClient.newConsumer(Schema.BYTES)
 		  .topic("my-topic")
		  .subscriptionName("my-subscription")
		  .subscriptionType(SubscriptionType.Shared)
         .deadLetterPolicy(DeadLetterPolicy.builder()
         		.maxRedeliverCount(maxRedeliveryCount)
         		.build())
         .subscribe();

The relationship between several retweets and DLQ

If DLQ is configured, data retries caused by AckTimeout, NegativeAck, or ReconsumeLater will trigger DLQ, which means that after the number of retries reaches the upper limit, it will be written to the DLQ topic.

The statistics of the number of retries are different:

Both AckTimeout and NegativeAck are counted through the Redelivery mechanism. After the SDK initiates a Redelivery request, the RedeliveryTracker on the Broker side will record the number of retries, and the message pushed to the Consumer will contain the RedeliveryCount field.

For RLQ, the number of repeated consumption is obtained from the RECONSUMETIMES attribute, which is generated in the Client and is also counted in the Client.

In general, Apache Pulsar provides a variety of message retweeting methods, and users can use them flexibly in combination with their own scenarios to meet their business needs.