1.kafka consumer processes
1.1. At startup or coordinator node failover, consumers send ConsumerMetadataRequest to bootstrap brokers any of a list of Brokers . In ConsumerMetadataResponse , the coordinator node which receives the location information corresponding to the consumption of the consumer group belongs.
1.2. Consumers connect the coordinator node, and send HeartbeatRequest . If the returned HeartbeatResponse returned IllegalGeneration error code, indicating the coordinator node is already initialized. Consumers will stop crawling data, submit offsets , send JoinGroupRequest to the coordinator node. In JoinGroupResponse , it receives the consumer should have topic-partitions new list of consumer groups and the current generation number. This time the consumer group management has been completed, consumers can begin to crawl data, and for its own partitions submit offsets .
1.3. If HeartbeatResponse no error is returned, consumers will last from its own partitions continue to crawl data list, this process will not be interrupted.
Coordinator coordination process node:
1. In the steady state, the coordinator node to track the health of each consumer group in each consumer by the failure detection protocol.
2. In the election and start coordinating node reads the list of consumer groups it manages, as well as from ZK to read the membership information for each consumer group. If there is no member information before, it will not do anything. Only when registering with the first consumer a consumer group to come in, the coordinator node to begin to work ( that is, to begin loading consumer information consumer group member ) .
3. When all group members before the coordinator node finished consumer group is fully loaded it is responsible for the list, it will respond to the request in the following return CoordinatorStartupNotComplete error code: HeartbeatRequest , OffsetCommitRequest , JoinGroupRequest . In this way consumers will retry over time ( until it is fully loaded, no error code is returned so far ) .
4. In the election or start, all consumers will consume the coordinator node group fault detection. According to the coordinator node is marked as a failure detection protocol Dead will be removed from the consumer group of consumers, this time coordinating node for the Dead consumer group belongs consumers to trigger a balancing operation( Consumer Dead later, the consumer has the partition need to balance other consumers ) .
5. When HeartbeatResponse return IllegalGeneration error code, balancing operation is triggered. Once all surviving consumers JoinGroupRequests re-register with the coordinator node, the coordinator node will be the latest partition ownership information in JoinGroupResponse communication between each consumer ( synchronous ) , and then to complete the balancing operation.
6. The coordinating node will track any consumer already registered topics of topic-partition changes. If it detects a topic new Partition , it will trigger the balancing operation. When creating a new topics will trigger the balancing operation, because consumers can topic register interest before it is created topics .
2. The consumer group usage scenarios
Kafka in the consumer group had used two scenarios:
2.1 "Queue Mode": In the same group of consumer consumption of all common theme of a message, but also to ensure that a message is only dealing with a consumer. A theme of all the partitions and all consumers do associate a consumer group: only one partition is associated with a consumer, its message will not be received by other consumers.
When a consumer is only the beginning, all partitions are assigned to it. When increasing the size of the message, we need to expand the number of consumers, horizontal scaling capabilities, can reach every consumer has been associated only one partition. Partition is greater than the number of consumers will be in an idle state, because there is no distribution of any partition.
2.2 "publish / subscribe model" : create different consumer groups means that a theme of the message is sent to all subscribers of its consumer groups, consumer groups and assign work together in accordance with the preceding scene. This is often because we have different application requirements, such as number of transactions, financial systems, ERP systems will consume it and risk monitoring also need to consume it. This achieves asynchronous transparent common data.
In both scenarios, the consumer group has an important function: Rebalancing . When a new consumers to join a group, if there is a valid partition (the number of consumers <= Partitions topic number of partitions), will begin a re-balancing of the distribution operations would have been associated with a (still its original consumer retain at least one partition) reassigned to new customers added. Similarly, when a consumer because of various reasons to leave the group, all of its partitions will be assigned to the rest of the consumer.
Subscribe (Auto) ASSIGN (manual)
aforesaid automatic dispensing means KafkaConsumer API in subscribe () method. This method requires you to set a mandatory consumer groups, consumers group.id parameter can not be null. And you do not need to deal with the allocation of partitions . To correspond subscribe () method . You can use manually specify which theme consumers read partition, then: ASSIGN () method. When you need precise control message processing load, but also can determine which partition what message this manually would be useful
The automatic submission API
[Hadoop @ H201 kafka_2.12-0.10.2.1] $ bin / kafka-topics.sh --create --zookeeper H201: 2181, H202: 2181, H203: 2181 --replication factor-2 - -partitions 3 --topic topic11
[hadoop @ H201 kkk] $ vi cc.java
Import org.apache.kafka.clients.consumer *;.
Import java.util.Properties;
Import java.util.concurrent.ExecutionException;
Import java.util.Arrays;
public class CC {
public static void main (String [] args) throws ExecutionException, {InterruptedException
the Properties the props = new new the Properties ();
// set the cluster address kafka
props.put ( "bootstrap.servers", "h201 : 9092, h202: 9092, H203: 9092 ");
// set the consumer group, the group name the custom, the name of the same group of consumers in a group
props.put (" group.id "," G11 ");
// open offset automatically submit
props. PUT ( "enable.auto.commit", "to true");
// automatic submission time interval
props.put ( "auto.commit.interval.ms", "1000");
// serializers
props.put ( "key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put ( "value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
// instantiate a consumer
KafkaConsumer <String, String> consumer = new new KafkaConsumer <> (props);
// consumers subscribe to a topic, you can subscribe to multiple topics
Consumer.subscribe (Arrays.asList ( "topic11"));
/ / stop cycle of death to take data from the broker
the while (to true) {
ConsumerRecords <String, String> Records = consumer.poll (100);
for (ConsumerRecord <String, String> Record: Records)
System.out.printf ( " D =% offset, Key =% S,% S =% n-value ", record.offset (), record.key (), record.value ());
}
}
}
[hadoop@h201 kkk]$ /usr/jdk1.8.0_144/bin/javac -classpath /home/hadoop/kafka_2.12-0.10.2.1/libs/kafka-clients-0.10.2.1.jar cc.java
[hadoop@h201 kkk]$ /usr/jdk1.8.0_144/bin/java cc
Explain:
Poll used to retrieve messages, poll (pulling)
consumer.poll (100): a pull data within 100ms
the Record: to store the message, record.value message content