Kafka is how to ensure data reliability and consistency
Students learn too much data should all know Kafka used to live , which is distributed message subscription system, there is a very good lateral scalability, real-time store huge amounts of data, stream data processing is the de facto standard middleware. This article describes Kafka how to ensure data reliability and consistency.
Article Directory
Data reliability
Kafka as a business-class messaging middleware, message reliability of importance can be imagined. Sending a message from Broker to herein Producter, Topic Leader Election partition replicas and reliability point of view on several data.
Topic partition replica
Before Kafka 0.8.0, Kafka is no concept of a copy, and then people will just give some important data stored by Kafka, because there is no copy of the data is likely to be lost. But as your business grows, more and more intense support copy function, so in order to ensure the reliability of the data, Kafka began to introduce a version from 0.8.0 partition replica (For details, see KAFKA-50 ). That each partition can be artificially configure several copies (such as creating a theme designated time replication-factor, can also be configured in Broker level default.replication.factor), usually set to 3.
Kafka can guarantee that in the event a single partition is ordered, zoning online (available) and offline (unavailable). Among the many partition copy of which is a copy of the Leader, the rest of the copy is a follower, all read and write operations are carried out through Leader, while follower will copy the data to the leader on a regular basis. When Leader hung up when one follower will again become the new Leader. By partitioning the copy, the data redundancy is introduced, but also provides the data reliability Kafka.
Kafka partition multi-core architecture is a copy of Kafka reliability assurance, the message is written to make multiple copies of Kafka in the event of a crash can still guarantee persistent messages.
Producer sending messages to the Broker
If we want to send a message to Kafka corresponding theme, we need to be completed by the Producer. Earlier we talked about Kafka theme corresponds to a multiple partitions, each of the following turn corresponds to multiple copies; in order to allow the user to set data reliability, Kafka provides a message confirmation mechanism Producer inside. That we can decide to send a message to several copies of the corresponding partition is considered the message sent by the success of the configuration. When defined by Producer acksspecification (version before 0.8.2.X parameter request.required.acks parameter settings, see KAFKA-3043 ). This parameter supports the following three values:
- acks = 0: means that if producers can send messages across the network, then that message has been successfully written to Kafka. In this case the error may still occur, such as sending an object can be serialized or non-NIC fails, but if it is a partition or the entire cluster offline for a long time is not available, it will not receive any errors. The speed at acks = 0 mode is very fast (which is why many benchmarks are based on this model), you can get amazing throughput and bandwidth utilization, but if this mode is selected, will be lost some messages.
- acks = 1: Leader means upon receipt if the message and write it to a file partition data (not necessarily synchronized to disk) will return an error response or acknowledgment. In this mode, if the normal Leader elections take place, producers will receive in the election of a LeaderNotAvailableException exception if the producer can properly handle the error, it will retry sending quiet interest, eventually the message will arrive safely at the new Leader There. However, there are still in this mode possible loss of data, such as message has been successfully written to the Leader, but before the message is copied to the copy follower Leader crashes.
- acks = all (the same as the meaning and request.required.acks = -1): means Leader before returning an error response or acknowledgment, waits until all synchronized copies of all received information quiet. If and
min.insync.replicasparameters together, you can decide before returning to confirm how many copies there are at least able to receive information quiet, producers will have to retry until the message is successfully submitted. But this is the slowest approach, as producers continue to send other messages before all copies have to wait to receive the current message.
According to the actual application scenarios, we set up different acks, in order to ensure the reliability of the data.
Further, Producer transmission message may also choose to sync (default, by producer.type=sync configuration) or asynchronous ( producer.type=async) mode. If set to asynchronous, although it will greatly improve messaging performance, but this increases the risk of losing data. If you need to ensure the reliability of the message must be producer.type set to sync.
Leader election
Before introducing Leader elections, let's take a look at the ISR (in-sync replicas) list. Each partition leader maintains a list of ISR, ISR list which is the follower Borker numbered copies, only to keep up with Leader of the follower a copy of which can be added to ISR, this is achieved by replica.lag.time.max.ms configuration parameters, reference may be made "article of understanding Kafka a copy of the replication mechanism " . Only the ISR members have been selected as a possible leader.
So when Leader hung up, and unclean.leader.election.enable=false in the case of, Kafka will select the first follower from the ISR list as the new Leader, because this partition has been committed with the latest news. This can ensure the reliability of the data by the already committed message.
In summary, in order to ensure the reliability of the data, we need at least a few parameters to configure it:
- Level producer: acks = all (or request.required.acks = -1), simultaneous synchronization mode producer.type = sync
- topic level: setting replication.factor> = 3, and min.insync.replicas> = 2;
- broker Grade: Incomplete Leader close election, that unclean.leader.election.enable = false;
Data consistency
Here presented data consistency Leader main point is that whether it is old or new Leader of the election, Consumer can read the same data. So Kafka is how to achieve it?
If you want to keep abreast of article Spark, Hadoop HBase or related to, welcome attention to the micro-channel public number: iteblog_hadoop
Suppose copy of the partition 3, where 0 is a copy of the Leader, and replica copies of a follower 2, and in which the list of ISR. Although the copy has been written 0 Message4, but only read to the Consumer Message2. Because all of the ISR Message2 are synchronized, only a High Water Mark above message was read Consumer support, depends on the minimum and the High Water Mark ISR list offset inside partition, corresponding to the FIG. 2 copies, this is very similar to barrel Theory.
The reason for this is not yet enough to copy a copy of the message is considered "unsafe", if the Leader crashes, another copy to become the new Leader, then these messages are likely to be lost. If we allow the consumer to read these messages might undermine consistency. Imagine, a consumer reads from the current Leader (copy 0) and processed Message4, this time Leader hung up, a copy of the election for the new Leader 1, this time to go to another consumer reads the message from the new Leader, I found this news does not actually exist, which leads to data inconsistency.
Of course, the introduction of the High Water Mark mechanism will lead to copy messages between Broker slow for some reason, the message will also reach consumers time becomes longer (because we will first wait for the completion of replication messages). The delay time may parameter replica.lag.time.max.ms configuration parameter that specifies the maximum message copy may be allowed while copying of a delay time.