- Why use kafka?
- How to ensure the data reliability of kafka
- Is Kafka's data on disk or in memory, why is it faster?
- Replica data synchronization strategy
- Troubleshooting
- How kafka transactions are implemented
- Why does Kafka not support read-write separation?
- Is Kafka's data on disk or in memory, why is it faster?
- Summarize
Why use kafka?
- Buffering and peak shaving: When there is a burst of traffic in the upstream data, the downstream may not be able to carry it, or there are not enough machines in the downstream to ensure redundancy. Kafka can play a buffering role in the middle, temporarily storing messages in Kafka, and downstream The service can be processed slowly at its own pace.
- Decoupling and extensibility: At the beginning of the project, the specific requirements cannot be determined. The message queue can be used as an interface layer to decouple important business processes. You only need to abide by the conventions and program for data to gain scalability.
- Redundancy: A one-to-many approach can be used. One producer publishes messages, which can be consumed by multiple services subscribed to the topic for multiple unrelated businesses.
- Robustness: The message queue can accumulate requests, so even if the consumer business dies in a short time, it will not affect the normal operation of the main business.
- Asynchronous communication: Many times, users do not want or need to process messages immediately. Message queues provide asynchronous processing mechanisms that allow users to put a message on the queue, but not process it immediately. Put as many messages as you want into the queue, and then process them when needed.
How to ensure the data reliability of kafka
In order to ensure that the data sent by the producer can be reliably sent to the specified topic, each partition of the topic needs to send an ack to the producer after receiving the data sent by the producer. Perform the next round of sending, otherwise resend the data. So lead to the ack mechanism. The ack response mechanism Kafka provides users with three reliability levels. Users can choose the following configurations according to the requirements of reliability and delay. acks parameter configuration:
- 0: The producer does not wait for the broker's ack. This operation provides the lowest latency. The broker will return as soon as it receives it and has not written to the disk. When the broker fails, data may be lost.
- 1: The producer waits for the ack from the broker, and returns the ack after the leader of the partition is successfully placed. If the leader fails before the follower is successfully synchronized, data will be lost.
- -1 (all): The producer waits for the ack from the broker, and returns the ack after all the leaders and followers of the partition are successfully placed. However, if the leader fails after the follower synchronization is completed and before the broker sends the ack, it will cause data duplication.
Is Kafka's data on disk or in memory, why is it faster?
Kafka uses disk storage. The speed is fast because: Sequential writing: Because the hard disk is a mechanical structure, each read and write will address -> write, where addressing is a "mechanical action", which is time-consuming. So hard drives "hate" random I/O and prefer sequential I/O. In order to improve the speed of reading and writing to the hard disk, Kafka uses sequential I/O. Memory Mapped Files: 20G data files can generally be represented in a 64-bit operating system. Its working principle is to directly use the operating system's Page to realize the direct mapping of files to physical memory. After completing the mapping, your operations on physical memory will be synchronized to the hard disk.
Kafka efficient file storage design: Kafka divides a large file in a topic into multiple small file segments. Through multiple small file segments, it is easy to periodically clear or delete the consumed files and reduce disk usage. The index information can quickly locate the message and determine the size of the response. By mapping all index metadata to memory (memory mapped file), the IO disk operation of segment file can be avoided. By sparse storage of index files, the space occupied by the metadata of index files can be greatly reduced.
Note: One of the ways Kafka solves the query efficiency is to segment the data file. For example, there are 100 Messages, and their offsets are from 0 to 99. Suppose the data file is divided into 5 segments, the first segment is 0-19, the second segment is 20-39, and so on, each segment is placed in a separate data file, and the data file is named after the small offset in the segment. In this way, when looking for the Message with the specified offset, the binary search can be used to locate which segment the Message is in. Indexing data files Data file segmentation makes it possible to find messages corresponding to offsets in a smaller data file, but this still requires sequential scans to find messages corresponding to offsets.
In order to further improve the efficiency of search, Kafka establishes an index file for each segmented data file. The file name is the same as that of the data file, except that the file extension is .index.
Replica data synchronization strategy
Reason for choosing the last one:
- Also in order to tolerate the failure of n nodes, the first scheme requires 2n+1 copies, while the second scheme only needs n+1 copies, and each partition of Kafka has a large amount of data, the first scheme will Cause a lot of data redundancy.
- Although the network latency of the second solution will be higher, the network latency will have less impact on Kafka.
If the ISR adopts the synchronization strategy of sending the copy of the ack after all synchronization is completed: ask the question: the leader receives the data, and all the followers start to synchronize the data, but one follower cannot synchronize with the leader due to some kind of failure. The leader has to wait until it completes synchronization before sending the ack. How to solve this problem? The leader maintains a dynamic in-sync replica set (ISR), which means a set of followers that keep pace with the leader. When the follower in the ISR completes data synchronization, the leader sends an ack to the follower. If the follower does not synchronize data with the leader for a long time, the follower will be kicked out of the ISR. The time threshold is set by the replica.lag.time.max.ms parameter. After the leader fails, a new leader is elected from the ISR.
Troubleshooting
LEO: refers to the maximum offset of each replica. HW: refers to the largest offset that the consumer can see, and the smallest LEO in the ISR queue. Kafka's consumption partition allocation strategy There are multiple consumers in a consumer group, and a topic has multiple partitions, so it will inevitably involve the problem of partition allocation, that is, to determine which consumer consumes Kafka for that partition, there are three allocation strategies, ** one It is RoundRobin and the other is Range. Higher versions also have a StickyAssignor strategy**
Moving ownership of a partition from one consumer to another is called rebalancing. When the following events occur, Kafka will perform a partition allocation: a new consumer is added to the same Consumer Group. Consumers leave the Consumer Group they currently belong to, including shuts down or crashes.
- Range partition allocation strategy
Range is for each topic (that is, one topic and one topic score). First, the partitions in the same topic are sorted according to the serial number, and the consumers are sorted in alphabetical order. Then divide the number of Partitions partitions by the total number of consumer threads to determine how many partitions each consumer thread consumes. If not, then the first few consumer threads will consume one more partition. Assuming n=number of partitions/number of consumers, m=number of partitions% number of consumers, then each of the first m consumers is allocated n+1 partitions, and each of the following (number of consumers-m) consumers is allocated n partitions.
If there are 10 partitions and 3 consumer threads, arrange the partitions by serial number 0, 1, 2, 3, 4, 5, 6, 7, 8, 9. The consumer threads are C1-0, C2-0, C2- 1 Then divide the number of partitions by the total number of consumer threads to determine how many partitions each consumer thread consumes. If the number of partitions cannot be divided, the first few consumers will consume one more partition.
In our example, we have 10 partitions, 3 consumer threads, 10/3 = 3, and if the division is inexhaustible, then the consumer thread C1-0 will consume one more partition, so the result of the final partition allocation It looks like this:
C1-0：0，1，2，3
C2-0：4，5，6
C2-1：7，8，9
If there are 11 partitions it will be:
C1-0：0，1，2，3
C2-0：4，5，6，7
C2-1：8，9，10
If we have two topics T1, T2, each with 10 partitions, the final allocation result will be as follows:
C1-0 ： T1 （0，1，2，3） T2 （0，1，2，3）
C2-0：T1（4，5，6） T2（4，5，6）
C2-1：T1（7，8，9） T2（7，8，9）
- RoundRobinAssignor partition assignment strategy
The principle of the RoundRobinAssignor strategy is to sort the partitions of all consumers in the consumer group and all topics subscribed by the consumers in lexicographical order, and then assign the partitions to each consumer one by one in a round-robin manner. There are two ways to use the RoundRobin strategy. The prerequisites must be met: the num.streams (number of consumer threads) of all consumers in the same consumer group must be equal; the topics subscribed by each consumer must be the same.
Join the topic-partitions group sorted by hashCode as follows
T1-5, T1-3, T1-0, T1-8, T1-2, T1-1, T1-4, T1-7, T1-6, T1-9
Our consumer threads are ordered as C1-0, C1-1, C2-0, C2-1
The result of the final partition assignment is:
Partition C1-0 will consume T1-5, T1-2, T1-6
Partition C1-1 will consume T1-3, T1-1, T1-9
Partition C2-0 will consume T1-0, T1-4
Partition C2-1 will consume T1-8, T1-7
- StickyAssignor partition assignment strategy
Kafka has introduced this allocation strategy since version 0.11.x. It has two main purposes: the allocation of partitions should be as uniform as possible, and the number of topic partitions allocated to consumers differs by at most one partition. The allocation remains the same. When the two conflict, the first objective takes precedence over the second.
In view of these two purposes, the specific implementation of the StickyAssignor strategy is much more complicated than the two assignment strategies of RangeAssignor and RoundRobinAssignor.
Suppose there are 3 consumers C0, C1, C2 in the consumption group, they all subscribe to 4 topics: t0, t1, t2, t3 and each topic has 2 partitions, that is to say, the entire consumption group subscribes to t0p0, t0p1 , t1p0, t1p1, t2p0, t2p1, t3p0, t3p1
The final allocation results of these 8 partitions are as follows:
Consumer C0: t0p0, t1p1, t3p0
Consumer C1: t0p1, t2p0, t3p1
Consumer C2: t1p0, t2p1
At first glance, this seems to be the same as the result allocated by the RoundRobinAssignor strategy. At this time, if consumer C1 leaves the consumer group, the consumer group will perform a rebalancing operation, and then the consumer partition will be reassigned.
If the RoundRobinAssignor strategy is adopted , the assignment results at this time are as follows:
Consumer C0: t0p0, t1p0, t2p0, t3p0
Consumer C2: t0p1, t1p1, t2p1, t3p1
As shown in the assignment results, the RoundRobinAssignor policy will re-polling assignments according to consumers C0 and C2.
And if the StickyAssignor strategy is used at this time , the assignment result is:
Consumer C0: t0p0, t1p1, t3p0, t2p0
Consumer C2: t1p0, t2p1, t0p1, t3p1
It can be seen that all the allocation results for consumers C0 and C2 in the previous allocation are retained in the allocation results, and the "burden" of the original consumer C1 is allocated to the remaining two consumers C0 and C2, and finally C0 and C2 distribution has remained balanced. If partition redistribution occurs, it is possible that the previous consumer and the newly assigned consumer are not the same for the same partition, and half of the processing of the previous consumer will be repeated in the newly assigned consumer. Again, this is obviously a waste of system resources. The StickyAssignor strategy, like the "sticky" in its name, makes the allocation strategy have a certain "stickiness", and makes the two allocations the same as possible, thereby reducing the consumption of system resources and the occurrence of other abnormal situations.
What has been analyzed so far is that the subscription information of consumers is the same. Let's take a look at the processing when the subscription information is different.
For example, there are 3 consumers in the same consumer group: C0, C1, C2 There are 3 topics in the cluster: t0, t1, t2
These three topics have 1, 2, and 3 partitions respectively, which means that there are 6 partitions of t0p0, t1p0, t1p1, t2p0, t2p1, and t2p2 in the cluster.
Consumer C0 subscribes to topic t0
Consumer C1 subscribes to topics t0 and t1
Consumer C2 subscribes to topics t0, t1 and t2
If the RoundRobinAssignor strategy is adopted at this time:
Consumer C0: t0p0
Consumer C1: t1p0
Consumer C2: t1p1, t2p0, t2p1, t2p2
If the StickyAssignor strategy is used at this time:
Consumer C0: t0p0
Consumer C1: t1p0, t1p1
Consumer C2: t2p0, t2p1, t2p2
At this time, the consumer C0 is out of the consumer group, then the allocation result of the RoundRobinAssignor strategy is
Consumer C1: t0p0, t1p1
Consumer C2: t1p0, t2p0, t2p1, t2p2
StickyAssignor strategy, then the assignment result is:
Consumer C1: t1p0, t1p1, t0p0
Consumer C2: t2p0, t2p1, t2p2
It can be seen that the StickyAssignor strategy retains the original 5 partition assignments in consumers C1 and C2: t1p0, t1p1, t2p0, t2p1, t2p2.
From the results, the StickyAssignor strategy is more excellent than the other two allocation strategies, and the code implementation of this strategy is also very complicated.
How kafka transactions are implemented
Kafka has introduced transaction support since version 0.11. Transactions can ensure that Kafka, based on the Exactly Once semantics, can produce and consume across partitions and sessions, and either all succeed or all fail.
- Producer Transactions
In order to implement cross-partition and cross-session transactions, it is necessary to introduce a globally unique Transaction ID, and bind the PID obtained by the Producer to the Transaction ID. In this way, when the Producer is restarted, the original PID can be obtained through the ongoing Transaction ID.
To manage transactions, Kafka introduces a new component, Transaction Coordinator. The Producer obtains the task status corresponding to the Transaction ID by interacting with the Transaction Coordinator. The Transaction Coordinator is also responsible for writing the transaction to an internal topic of Kafka, so that even if the entire service is restarted, since the transaction state is saved, the transaction state in progress can be restored and the process can continue.
- Consumer transaction
For the Consumer, the guarantee of the transaction will be relatively weak, especially the information of the Commit cannot be guaranteed to be accurately consumed. This is because Consumers can access any information through offset, and different Segment Files have different life cycles, so the messages of the same transaction may be deleted after restarting. Exactly Once semantics sets the ACK level of the server to -1 to ensure that no data is lost between the Producer and the Server, that is, At Least Once semantics. In contrast, setting the server ACK level to 0 can ensure that each message of the producer will only be sent once, that is, the At Most Once semantics.
Before Kafka version 0.11, there was nothing that could be done about this. It could only ensure that the data was not lost, and then the downstream consumers would globally deduplicate the data.
In the case of multiple downstream applications, each needs to be deduplicated globally, which has a great impact on performance. The 0.11 version of Kafka introduced a major feature: idempotency. Turn on idempotency enable.idempotence=true. The so-called idempotency means that no matter how many times the Producer sends repeated data to the Server, the Server side will only persist one. Idempotency combined with At Least Once semantics constitute Kafka's Exactly Once semantics. That is: At Least Once + idempotency = Exactly Once Kafka's idempotency implementation is actually to replay the original downstream needs to the data upstream. A Producer with idempotency enabled will be assigned a PID during initialization, and messages sent to the same Partition will be accompanied by a Sequence Number. The Broker side will cache <PID, Partition, SeqNumber>. When a message with the same primary key is submitted, the Broker will only persist one. However, the PID will change after restarting, and different Partitions also have different primary keys, so idempotency cannot guarantee Exactly Once across partitions and sessions. Supplementary, what is the Exactly Once semantics in stream computing? Take flink as an example
- source: Use the data source that executes ExactlyOnce, such as kafka, etc.
Internally use FlinkKafakConsumer and enable CheckPoint, the offset will be saved to StateBackend, and the offset will be written to topic by default, that is, _ consumer_offsets Flink sets
CheckepointingModel.EXACTLY_ONCE
Why does Kafka not support read-write separation?
In Kafka, the operations of producers to write messages and consumers to read messages all interact with the leader copy, thus realizing a main-write-main-read production and consumption model. Kafka does not support master-write-slave read, because the master-write-slave read has two obvious shortcomings: data consistency problem: data from the master node to the slave node will inevitably have a delay time window, this time window will lead to The data between the master and slave nodes is inconsistent. At a certain moment, the value of A data in the master node and the slave node is X, and then the value of A in the master node is modified to Y, then the application reads the A data in the slave node before the change is notified to the slave node The value of is not the latest Y, which causes the problem of data inconsistency.
Delay problem: Similar to Redis, the process of data writing from the master node to the synchronization to the slave node needs to go through the stages of network → master node memory → network → slave node memory, and the whole process will take a certain amount of time. In Kafka, master-slave synchronization is more time-consuming than Redis. It needs to go through the stages of network → master node memory → master node disk → network → slave node memory → slave node disk. For latency-sensitive applications, the master-write-slave-read function is not very suitable.
Is Kafka's data on disk or in memory, why is it faster?
Kafka uses disk storage. Fast because: sequential writes. Because the hard disk is a mechanical structure, each read and write will address -> write, where addressing is a "mechanical action", which is time-consuming. So hard drives "hate" random I/O and prefer sequential I/O. In order to improve the speed of reading and writing to the hard disk, Kafka uses sequential I/O. Memory Mapped Files: 20G data files can generally be represented in a 64-bit operating system. Its working principle is to directly use the operating system's Page to realize the direct mapping of files to physical memory. After completing the mapping, your operations on physical memory will be synchronized to the hard disk.
Kafka's efficient file storage design: Kafka divides a large parition file in a topic into multiple small file segments. Through multiple small file segments, it is easy to periodically clear or delete the consumed files and reduce disk usage.
The index information can quickly locate the message and determine the size of the response. By mapping all index metadata to memory (memory-mapped files), IO disk operations for segment files can be avoided. By sparse storage of index files, the space occupied by the metadata of index files can be greatly reduced.
Summarize
One of Kafka's means of solving query efficiency is to segment data files, for example, there are 100 messages, and their offsets are from 0 to 99. Suppose the data file is divided into 5 segments, the first segment is 0-19, the second segment is 20-39, and so on, each segment is placed in a separate data file, and the data file is named after the small offset in the segment. In this way, when looking for the Message with the specified offset, the binary search can be used to locate which segment the Message is in. Indexing data files Data file segmentation makes it possible to find messages corresponding to offsets in a smaller data file, but this still requires sequential scans to find messages corresponding to offsets. In order to further improve the efficiency of search, Kafka establishes an index file for each segmented data file. The file name is the same as that of the data file, except that the file extension is .index.
Original link:
https://mp.weixin.qq.com/s?__biz=Mzg2ODU1MDkwMw==&mid=2247486303&idx=1&sn=271ce974d598b890207218a136d60089&utm_source=tuicool&utm_medium=referral
