Kafka uses a variety of mechanisms to ensure that messages can be consumed, so as to achieve data reliability and persistence.
Here are some common methods and strategies to improve message reliability:
- Replication mechanism: Kafka uses the concepts of partitions and replicas. Each partition can have multiple copies, distributed on different Brokers. When a message is written to a partition, it is replicated to all replicas of that partition. In this way, even if a Broker fails, data can still be read from other replicas to ensure data reliability.
- Message confirmation mechanism: After a consumer successfully consumes a message, it will send a message confirmation mechanism, and sometimes mark the message as consumed
- ISR (In-Sync Replicas) mechanism: ISR refers to the set of replicas that are kept in sync with the Leader replica. Kafka ensures that only the copy in the ISR can become the leader, which ensures that the leader copy will wait for other copies in the ISR to complete synchronization after the message is written, thus ensuring the reliability of the message.
- Consumer retry: If the message consumption fails, the consumer can implement a retry strategy and try to re-consume the failed message until it succeeds