Article directory
Kafka cluster goals
- High concurrency
- High availability
- Dynamic scaling
How to estimate Kafka cluster size
Throughput:
Clustering can increase the ability to handle requests. The performance deficiency of a single Broker can be solved by expanding the broker.
disk space:
If a cluster has 10TB of data to retain and each broker can store 2TB, then at least 5 brokers are needed. If data replication is enabled, double the space is required, so this cluster requires 10 brokers.
Kafka cluster construction practice
Use two Linux servers: one 192.68.10.7 and one 192.168.10.8
Configuration information modification of 192.68.10.7
Configuration information modification of 192.168.10.8
Kafka cluster principle
Membership and Controllers
The controller is actually a broker, but in addition to the functions of a general broker, it is also responsible for the election of partition leaders.
When the controller discovers that a broker has joined the cluster, it uses the broker ID to check whether the newly joined broker contains a copy of the existing partition. If so, the controller sends the change notification to the newly added broker and other brokers, and the replica on the new broker starts replicating messages from the Leader.
in short. Kafka uses Zookeeper's ephemeral nodes to elect controllers and notify controllers when nodes join or exit the cluster. The controller is responsible for partition leader election when nodes join or leave the cluster.
It is obvious from the two startup logs below that the server 192.168.10.7 is the controller.
Cluster working mechanism
Replication functionality is at the heart of Kafka's architecture. In Kafka's documentation, Kafka describes itself as "a distributed, partitionable, replicable commit log service."
Replication is so critical because it ensures Kafka's availability and durability even when individual nodes fail.
Kafka uses topics to organize data. Each topic is divided into several partitions, and each partition has multiple copies. Those replicas are stored on brokers, and each broker can store hundreds or thousands of replicas belonging to different topics and partitions.
replication-factor parameter
Create an erdan topic, the replication factor is 2, and the number of partitions is 2
./kafka-topics.sh --bootstrap-server 192.168.10.7:9092 --create --topic erdan --replication-factor 2 --partitions 2
replication-factor is used to set the number of replicas of the topic. Each topic can have multiple replicas, and the replicas are located on different brokers in the cluster, which means that the number of replicas cannot exceed the number of brokers.
In partition0, broker1 (broker.id =0) is the leader, and broker2 (broker.id =1) is the follower replica.
In partition1, broker2 (broker.id =1) is the leader, and broker1 (broker.id =0) is the follower replica.
Each partition has a copy of the leader. To ensure consistency, all producer and consumer requests go through this copy.
Copies other than the leader are follower copies. Follower replicas do not process requests from clients. Their only task is to copy messages from the leader and maintain a consistent state with the leader. If the leader crashes, one of the followers will be promoted to the new leader.
auto.leader.rebalance.enable parameter
Whether to allow regular Leader election.
Setting its value to true means that Kafka is allowed to periodically conduct Leader re-election for some Topic partitions. Of course, this re-election is not carried out mindlessly. It must meet certain conditions before it can occur.
For example, Leader A has been performing very well, but if auto.leader.rebalance.enable=true, then Leader A may be forcibly resigned and replaced by Leader B after a period of time.
Know that changing the Leader is very expensive. All clients that originally sent requests to A must switch to sending requests to B. Moreover, there is essentially no performance benefit from changing the Leader. Therefore, it is recommended to set this parameter in the production environment. false.
Cluster message production
reliable producer
3 different acknowledgment modes for sending acknowledgment mechanism.
acks=0 means that if the producer can send the message out over the network, the message is considered to have been successfully written to Kafka.
acks=1 means that if the leader receives the message and writes it to the partition data file (not necessarily synchronized to disk), it will return an acknowledgment or error response.
acks=all means that the leader will wait (min.insync.replicas) for all synchronized replicas to receive the message before returning an acknowledgment or error response.
ISR（In-sync Replicas）
Kafka's data replication is based on Partition. Data replication between multiple backups is completed by the Follower pulling data from the Leader. From this point of view, it is a bit like the Master-Slave solution. The difference is that Kafka is neither a completely synchronous replication nor a completely asynchronous replication, but a dynamic replication solution based on ISR.
ISR, also known as In-Sync Replica. The leader of each Partition maintains such a list, which contains all replicas synchronized with it (including the leader itself). Each time data is written, only when all Replica in the ISR are copied, the Leader will set it to Commit, and it can be consumed by the Consumer.
This solution is very close to synchronous replication. But the difference is that this ISR is dynamically maintained by the Leader. If the Follower cannot "keep up" with the Leader, it will be removed from the ISR by the Leader. After it "keeps up" with the Leader again, it will be added to the ISR by the Leader again. Each time the ISR is changed, the Leader will persist the latest ISR to Zookeeper.
As for how to determine whether a Follower is "keeping up" with the Leader, the strategies of different versions of Kafka are slightly different.
Starting from version 0.9.0.0, replica.lag.max.messages has been removed, so the Leader no longer considers the number of messages behind the Follower. In addition, the Leader will not only determine whether the Follower sends a Fetch request to it within the replica.lag.time.max.ms time, but will also consider whether the Follower is synchronized with it within that time.
Reasons to use ISR scheme
Because the Leader can remove Followers that cannot be synchronized with it in time, compared with synchronous replication, it can avoid the slowest Follower from slowing down the overall speed, that is, ISR improves system availability.
All Followers in the ISR contain all committed messages, and only committed messages will be consumed by the Consumer. Therefore, from the Consumer's perspective, all Replica in the ISR are always in a synchronized state, thus incompatible with the asynchronous replication scheme. Improved data consistency compared to
ISR related configuration instructions
The min.insync.replicas parameter of the Broker specifies the minimum length of the ISR required by the Broker. The default value is 1. That is to say, in the extreme case, the ISR can only include the Leader. But if the Leader goes down at this time, the Partition will be unavailable and availability cannot be guaranteed. min.insync.replicas is a balance between kafka system availability and data reliability!
Only messages synchronized by all Replica in the ISR are committed. However, when the Producer publishes data, the Leader does not need all Replica in the ISR to synchronize the data before confirming receipt of the data. Producer can use the acks parameter to specify the minimum number of Replicas required to confirm receipt of the message before the message is deemed to be successfully sent. The default value of acks is 1, that is, the Leader immediately tells the Producer to receive the message after receiving the message. At this time, if the Leader crashes before the message in the ISR is copied, the message will be lost. And if this value is set to 0, after the Producer sends the data, it will immediately think that the data was sent successfully without any waiting. In fact, the data may fail to be sent, and the Producer's Retry mechanism will not take effect.
A more recommended approach is to set acks to all or -1. At this time, only when all Replica in the ISR receive the data (that is, the message is Commit), the Leader will tell the Producer that the message was sent successfully, thus ensuring that no There will be unknown data loss.