A configuration diagram above topic three partition. Partition1 two offset: 0 and 1. Partition2 There are four offset. Partition3 have an offset. Machine id id and a copy of a copy of where exactly the same.
If the number of copies of a topic is 3, then Kafka will create three identical copies for each partition in the cluster. a memory broker in each cluster or a plurality of partition. Multiple producer and consumer at the same time production and consumption data.
One. Broker
Kafka cluster includes one or more servers, called server nodes broker.
Stored in topic data broker. If a topic partition there are N, N clusters have a broker, each broker then stored in a partition in the topic.
If a topic has N partition, there is the cluster (N + M) th broker, which then stores the N broker a partition in the topic, the remaining partition the M broker does not store data in the topic.
If there is a topic of the N Partition, broker cluster number less than N, then a broker that stores a plurality of topic or partition. In the actual production environment, try to avoid this from happening, this situation is likely to lead to Kafka cluster data is not balanced.
two. Topic
Each cluster issued to Kafka's message has a category, the category is called Topic. (Topic different physically separate storage of messages, the message is logically a Topic although stored on one or more of the message broker, but the user to specify a Topic to production or consumption data without concern for where the data are stored in)
Similar to the name of the database table
three. partition
Topic data is partitioned into one or more partition. Each topic has at least one partition. Data in each partition is stored using a plurality of segment files. Data partition is ordered, data between different partition missing sequence data. If there are multiple topic partition, when consumption data can not guarantee the order of the data. In scene requires strict order to ensure that the message of consumption, the number of partition needs to be set to 1.
four. producer
Producers ie, the issuer data, the role of the news release to the topic of Kafka. After the broker receives the message sent by the producer, the message broker is added to the current segment for the file of additional data. Message sent by the producer, stored in a partition, a producer can specify stored data partition.
Fives. Consumer
Consumers can read the data from the broker. Consumers can consume data from multiple topic is.
六。Consumer Group
Each belongs to a particular Consumer Consumer Group (Consumer may be specified for each group name, if the group name specified in the Default group is).
Seven. Leader
Each partition has a plurality of copies, wherein there is only one as Leader, Leader partition is currently responsible for reading and writing data.
Eight. Follower
Follower follows the Leader, all write requests are routed through Leader, data changes will be broadcast to all Follower, Leader and Follower maintain data synchronization. If the Leader fails, from the Follower elect a new Leader. When the Follower and Leader hang up, stuck or slow sync, leader will be deleted from the follower "in sync replicas" (ISR) list, re-create a Follower.