Kafka message queue installation and use (two)

Three. Kafka architecture in-depth

3.1 Kafka workflow and file storage mechanism

In Kafka, messages are classified by topic. Producers produce messages and consumers consume messages, which are all topic-oriented.
Topic is a logical concept, and partition is a physical concept. Each partition corresponds to a log file, and the log file stores the data produced by the producer. The data produced by the Producer will be continuously appended to the end of the log file, and each piece of data has its own offset. Each consumer in the consumer group will record in real time which offset they have consumed, so that when the error is restored, they can continue to consume from the last location.

Since the messages produced by the producer are constantly appended to the end of the log file, in order to prevent the log file from being too large and causing inefficient data positioning, Kafka adopts a fragmentation and indexing mechanism to divide each partition into multiple segments. Each segment corresponds to two files-".index" file and ".log" file. These files are located in a folder, and the naming rule of the folder is: topic name + partition number. For example, if the topic first has three partitions, the corresponding folders are first-0, first-1, and first-2.

00000000000000000000.index
00000000000000000000.log
00000000000000170410.index
00000000000000170410.log
00000000000000239430.index
00000000000000239430.log

The index and log files are named after the offset of the first message of the current segment. The following figure shows the structure diagram of index file and log file.

The ".index" file stores a large amount of index information, the ".log" file stores a large amount of data, and the metadata in the index file points to the physical offset address of the message in the corresponding data file.

3.2 Kafka producer

3.2.1 Partitioning strategy

1) Reasons for partition

• It is convenient to expand in the cluster. Each Partition can be adjusted to adapt to the machine where it is located, and a topic can be composed of multiple Partitions, so the entire cluster can adapt to data of any size;
• Concurrency can be improved because it can be read and written in units of Partition.

2) The principle of partitioning
We need to encapsulate the data sent by the producer into a ProducerRecord object.

• In the case of specifying the partition, directly use the specified value as the partition value;
• If the partition value is not specified but there is a key,
  take the remainder of the hash value of the key and the partition number of the topic to obtain the partition value;
• When there is neither a partition value nor a key value, an integer is randomly generated during the first call (increment on this integer for each subsequent call), and the remainder of this value and the total number of partitions available for the topic are taken to obtain the partition value. It is the often-called round-robin algorithm.

3.2.2 Data reliability guarantee

In order to ensure that the data sent by the producer can be reliably sent to the specified topic, after each partition of the topic receives the data sent by the producer, it needs to send an ack (acknowledgement confirmation) to the producer. If the producer receives an ack, it will Send the next round, otherwise resend the data.

Kafka chose the second scheme for the following reasons:
1. In order to tolerate the failure of n nodes, the first scheme requires 2n+1 copies, while the second scheme only requires n+1 copies, and Kafka’s Each partition has a large amount of data, and the first scheme will cause a large amount of data redundancy.
2. Although the network delay of the second scheme will be relatively high, the network delay has less impact on Kafka.
2) After ISR
adopts the second scheme, imagine the following scenario: the leader receives data, and all followers start to synchronize data, but there is one follower, because of some kind of failure, it can not synchronize with the leader for a long time, then the leader has to wait Go on until it completes synchronization before sending an ack. How to solve this problem?
The leader maintains a dynamic in-sync replica set (ISR), which means a follower set that is synchronized with the leader. When the follower in the ISR completes data synchronization, the leader will send an ack to the follower. If the follower does not synchronize data with the leader for a long time, the follower will be kicked out of the ISR. The time threshold is set by the replica.lag.time.max.ms parameter. After the leader fails, a new leader will be elected from the ISR.
3) ack response mechanism
For some less important data, the reliability requirements of the data are not very high, and a small amount of data loss can be tolerated, so there is no need to wait for all the followers in the ISR to receive successfully.
Therefore, Kafka provides users with three reliability levels. Users can choose the following configuration according to the requirements of reliability and delay.
acks parameter configuration:
acks:

• 0: The producer does not wait for the ack of the broker. This operation provides the lowest delay. The broker returns as soon as it receives it and has not written to the disk. When the broker fails, data may be lost;
• 1: The producer waits for the ack of the broker, and the leader of the partition returns ack after successfully placing the disk. If the leader fails before the follower synchronization is successful, data will be lost;
  
• 1 (all): The producer waits for the ack of the broker, leader and follower of the partition before returning to the ack. But if the leader fails after the follower synchronization is completed and before the broker sends an ack, it will cause data duplication.
  
  4) Details of troubleshooting
  
• LEO: Refers to the maximum offset of each copy.
• HW: Refers to the largest offset that consumers can see, and the smallest LEO in the ISR queue.

(1) Follower failure When a follower fails, the
follower will be temporarily kicked out of the ISR. After the follower recovers, the follower will read the last HW recorded by the local disk and intercept the part of the log file higher than HW, starting from HW Synchronize to the leader. After the follower's LEO is greater than or equal to the partition's HW, that is, after the follower catches up with the leader, you can rejoin the ISR.
(2) Leader failure After the
leader fails, a new leader will be selected from the ISR. After that, to ensure data consistency between multiple copies, the remaining followers will first set their log files higher than the HW Cut off, and then synchronize data from the new leader.
Note: This can only guarantee data consistency between copies, but does not guarantee that data will not be lost or duplicated.

3.2.3 Exactly Once Semantics

Setting the ACK level of the server to -1 can ensure that no data will be lost between the Producer and the Server, that is, the semantics of At Least Once. In contrast, setting the server ACK level to 0 can ensure that each message of the producer will only be sent once, that is, the semantics of At Most Once.

At Least Once can guarantee that the data is not lost, but it cannot guarantee that the data is not repeated; On the contrary, At Most Once can guarantee that the data is not repeated, but it cannot guarantee that the data is not lost. However, for some very important information, such as transaction data, downstream data consumers require that the data is neither duplicated nor lost, that is, Exactly Once semantics. Before version 0.11, Kafka was powerless. It could only ensure that the data was not lost, and then downstream consumers would globally de-duplicate the data. In the case of multiple downstream applications, each needs to be individually deduplicated globally, which has a great impact on performance.

Kafka version 0.11 introduced a major feature: idempotence. The so-called idempotence means that no matter how many times the Producer sends duplicate data to the server, the server will only persist one. Idempotence combined with At Least Once semantics constitutes Kafka's Exactly Once semantics. That is:
At Least Once + idempotence = Exactly Once
To enable idempotence, you only need to set enable.idompotence in the parameter of Producer to true. The realization of Kafka's idempotence is actually to de-replace the original downstream needs to the data upstream. The Producer with idempotence turned on will be assigned a PID when it is initialized, and the message sent to the same Partition will be accompanied by a Sequence Number. The Broker will cache <PID, Partition, SeqNumber>. When a message with the same primary key is submitted, the Broker will only persist one.
But the PID will change when restarted, and different Partitions also have different primary keys, so idempotence cannot guarantee Exactly Once across partitions and sessions.

3.3 Kafka consumers

3.3.1 Consumption mode

The consumer uses the pull mode to read data from the broker.
The push model is difficult to adapt to consumers with different consumption rates, because the message sending rate is determined by the broker. Its goal is to deliver messages as quickly as possible, but this can easily cause consumers to be too late to process messages. Typical manifestations are denial of service and network congestion. The pull mode can consume messages at an appropriate rate according to the consumer's consumption capacity.

The disadvantage of the pull mode is that if Kafka has no data, consumers may fall into a loop and return empty data all the time. In response to this, Kafka consumers will pass in a duration parameter timeout when consuming data. If there is currently no data available for consumption, the consumer will wait for a period of time before returning. This period of time is called timeout.

3.3.2 Partition allocation strategy

There are multiple consumers in a consumer group, and a topic has multiple partitions, so the allocation of partitions is bound to be involved, that is, to determine which consumer consumes that partition.
Kafka has two allocation strategies, one is RoundRobin and the other is Range.
1) RoundRobin

2）Range

3.3.3 Offset maintenance

Since the consumer may experience failures such as power outages and downtime during the consumption process, after the consumer recovers, it needs to continue to consume from the location before the failure, so the consumer needs to record in real time which offset it consumes so that it can continue to consume after the failure recovery.

Before Kafka 0.9 version, the consumer saves the offset in Zookeeper by default. Starting from version 0.9, the consumer saves the offset in a built-in Kafka topic by default, which is __consumer_offsets.
1) Modify the configuration file consumer.properties

exclude.internal.topics=false

2) Read offset (version after 0.11.0.0 (inclusive))

bin/kafka-console-consumer.sh --topic __consumer_offsets --
zookeeper hadoop102:2181 --formatter 
"kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageForm
atter" --consumer.config config/consumer.properties --frombeginning

3.3.4 Consumer Group Case

1) Demand: To test consumers in the same consumer group, only one consumer can consume at a time.

2) Case practice

(1) Change the group.id property in the config/consumer.properties configuration file to any group name in hadoop102 hadoop103.

[atguigu@hadoop103 config]$ vi consumer.properties
group.id=atguigu

(2) Start consumers separately on hadoop102 and hadoop103

[atguigu@hadoop102 kafka]$ bin/kafka-console-consumer.sh \
--zookeeper hadoop102:2181 --topic first --consumer.config 
config/consumer.properties
[atguigu@hadoop103 kafka]$ bin/kafka-console-consumer.sh --
bootstrap-server hadoop102:9092 --topic first --consumer.config 
config/consumer.properties

(3) Start the producer on hadoop104

[atguigu@hadoop104 kafka]$ bin/kafka-console-producer.sh \
--broker-list hadoop102:9092 --topic first
>hello world

(4) Check the receivers of hadoop102 and hadoop103.

同一时刻只有一个消费者接收到消息。

3.4 Kafka efficiently read and write data

1) Write the
production data of Kafka's producer on disk sequentially . To write to the log file, the writing process is to append to the end of the file, which is sequential writing. There is data on the official website that the same disk can be written up to 600M/s in sequence, but only 100K/s in random. This is related to the mechanical mechanism of the disk. The reason why sequential writing is fast is because it saves a lot of head addressing time.
2) Zero-copy technology For

details, please refer to the blog: https://www.jianshu.com/p/1c27da322767 .

3.5 The role of Zookeeper in Kafka

A broker in the Kafka cluster will be elected as the Controller, responsible for managing the online and offline of the broker in the cluster, the allocation of partition copies of all topics, and leader election.
Controller's management work is dependent on Zookeeper.
The following is the leader election process of the partition:

3.6 Kafka transaction

Kafka has introduced transaction support since version 0.11. Transactions can ensure that Kafka is based on Exactly Once semantics. Production and consumption can cross partitions and sessions, and either all succeed or all fail.

1. Producer transactions
   In order to implement cross-partition and cross-session transactions, it is necessary to introduce a globally unique Transaction ID and bind the PID obtained by the Producer with the Transaction ID. In this way, when the Producer is restarted, the original PID can be obtained through the ongoing Transaction ID.
   To manage Transaction, Kafka introduced a new component Transaction Coordinator. The Producer obtains the task status corresponding to the Transaction ID by interacting with the Transaction Coordinator. The Transaction Coordinator is also responsible for writing all transactions to an internal topic of Kafka, so that even if the entire service restarts, as the transaction status is saved, the transaction status in progress can be restored and proceed.
2. Consumer transaction The
   above transaction mechanism is mainly considered from the producer side. For Consumer, the transaction guarantee will be relatively weak, especially when the Commit information cannot be guaranteed to be accurately consumed. This is because Consumer can access any information through offset, and different Segment File life cycles are different, and messages of the same transaction may be deleted after restart.