foreword
With the development of microservices, Kafka message middleware is an important means of asynchronous decoupling and traffic peak reduction. At the same time, how to ensure that messages are not lost has also become a problem that must be solved. The following seven suggestions will help you solve it.
suggestion
Set the callback function when the message producer sends a message
Setting the callback function is actually a message confirmation mechanism, which has the following functions:
- Whether the listening message successfully reaches the broker
- Record the message log to mongo, which is easy to locate the problem
- Some message compensation mechanisms can be appropriately done, such as resending failed messages
Set the acks parameter
acks is a parameter of the Producer that represents the definition of "committed" messages. The values that can be set are all, 0, 1, -1. Can be set in combination with actual business
#procedure要求leader在考虑完成请求之前收到的确认数，用于控制发送记录在服务端的持久化，其值可以为如下：
#acks = 0 如果设置为零，则生产者将不会等待来自服务器的任何确认，该记录将立即添加到套接字缓冲区并视为已发送。在这种情况下，无法保证服务器已收到记录，并且重试配置将不会生效（因为客户端通常不会知道任何故障），为每条记录返回的偏移量始终设置为-1。
#acks = 1 这意味着leader会将记录写入其本地日志，但无需等待所有副本服务器的完全确认即可做出回应，在这种情况下，如果leader在确认记录后立即失败，但在将数据复制到所有的副本服务器之前，则记录将会丢失。
#acks = all 这意味着leader将等待完整的同步副本集以确认记录，这保证了只要至少一个同步副本服务器仍然存活，记录就不会丢失，这是最强有力的保证，这相当于acks = -1的设置。
spring.kafka.producer.acks=all
复制代码Set the retries parameter
A larger value is recommended. The same is the parameter of the Producer. When network jitter occurs, message sending may fail. At this time, the Producer configured with retries can automatically retry sending messages to avoid message loss as much as possible.
spring.kafka.producer.retries=3
复制代码设置unclean.leader.election.enable = false。
This is a parameter on the Broker side. In the iteration of the kafka version, the community has repeatedly modified its default value, which was controversial before. It controls which Brokers are eligible to run for the division's leader. If a broker is too far behind the original leader, it will cause message loss once it becomes the new leader. Therefore, it is generally necessary to set this parameter to false.
unclean.leader.election.enable = false
复制代码Set replication.factor >= 3.
This is also a parameter on the Broker side. Save multiple copies of the message redundancy, not much to explain.
replication.factor >= 3
复制代码Set min.insync.replicas > 1.
Broker-side parameter, which controls how many replicas the message must be written to before it is considered "committed". Set to greater than 1 to improve message durability. Do not use the default value of 1 in a production environment. Make sure replication.factor > min.insync.replicas. If the two are equal, then as long as one replica is offline, the entire partition won't work. It is recommended to set replication.factor = min.insync.replicas + 1.
Make sure that the message consumption is completed and then submitted.
There is a parameter enable.auto.commit on the consumer side, it is best to set it to false and handle the commit update of the offset by itself.