2.1生产者消息顺序保证

对于需要保证顺序的消息,在Producer发布消息时,需要保证分发到同一Partition,并且设置参数max.in.flight.requests.per.connection=1。同时,每个Partition只会被consumer group中的一个consumer消费。如此便可严格保证Kafka消息的顺序。

详细解释:

Kafka可以保证同一个分区里的消息是有序的。也就是说,如果生产者按照一定的顺序发送消息,broker就会按照这个顺序把它们写入分区,消费者也会按照同样的顺序读取它们。

如果把生产者的retries设为非零整数,同时把max.in.flight.requests.per.connection设为比1大的数,那么第一批次消息写入失败,而第二个批次写入成功,broker会重试写入第一个批次。如果此时第一个批次也写入成功,那么两个批次的顺序就反过来了。

一般来说,如果某些场景要求消息是有序的,那么消息写入成功也是很关键的,所以不建议把retries设为0。可以把max.in.flight.requests.per.connection设为1,这样在生产者尝试发送第一批消息时,就不会有其他的消息发送给broker。不过这样会严重影响生产者的吞吐量,所以只有在对消息的顺序有严格要求的情况下才能这样做。

参数说明:

retries (default: 0)

Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max.in.flight.requests.per.connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first.

max.in.flight.requests.per.connection (default: 5)

The maximum number of unacknowledged requests the client will send on a single connection before blocking. Note that if this setting is set to be greater than 1 and there are failed sends, there is a risk of message re-ordering due to retries (i.e., if retries are enabled).

参考

1.Kafka权威指南

2.Kafka documentation

猜你喜欢

转载自www.cnblogs.com/pugongying017/p/9616154.html