The study notes Kafka Kafka Consumer Design Analysis

0x00 Summary

　　This paper describes Kafka High Level Consumer, Consumer Group, Consumer Rebalance, Low Level Consumer realization of semantics, and application scenarios. As well as future versions of the redesign of the High Level Consumer - Use Consumer Coordinator to address issues such as Split Brain and Herd.

0x01 High Level Consumer

　　Many times, the client just wants to read data from Kafka, do not care about the offset process the message. We also hope to provide some semantics, such as the same message was only one Consumer consumption (unicast) or all Consumer Consumption (broadcast). Therefore, Kafka Hight Level Consumer provides a high level of abstraction consumption data from Kafka, in order to block out the details and provide rich semantics.

1.1 Consumer Group

　　Offset last message from the High Level Consumer stored in a read Partition in Zookeeper ( Kafka used to live from the 0.8.2 version began to support the same time offset stored in Zookeeper with the offset stored in a dedicated Kafka Topic in ). The offset program based on the client to provide the name of Kafka to save, the name is called Consumer Group. Consumer Group is a global whole cluster of Kafka, instead of a Topic. Each instance of a High Level Consumer belong to a Consumer Group, without specifying the default Group belongs.
　　Consumer Zookeeper the relevant nodes as shown below

　　Many traditional Message Queue will be consumed in the message will be removed after the news, on the one hand to avoid duplication of consumption, on the other hand can guarantee Queue length is relatively short, and improve efficiency. And as mentioned above, Kafka does not delete messages consumed, in order to achieve the traditional Message Queue message was consumed only once semantics, Kafka ensure that each message in the same Consumer Group will be consumed in one Consumer. Message Queue with the traditional difference is that, Kafka also allows different Consumer Group consume the same message at the same time, this feature provides support for a diversified processing messages.

　　In fact, one of Kafka's design philosophy is to also provide off-line processing and real-time processing. According to this feature, you can use this real-time streaming Storm system messages in real-time online processing, while using Hadoop batch of such systems off-line processing, real-time while also back up data to another data center, the need to ensure that only Consumer three operations can be used in different Consumer Group. The following figure shows a simplified deployment model in Kafka on LinkedIn.

　　In order to more clearly display the characteristics of Kafka Consumer Group, the author conducted a test. Create a Topic (named topic1), then create an instance belonging Consumer group1 and creates three instances belonging Consumer group2, and then sends topic1 respectively by 1,2,3 Producer Key message. Consumer belonging found group1 received all three messages, while in group2 were three Consumer 1,2,3 Key received message, as shown in FIG.

Note: The image above each black areas represent a Consumer instances, each instance is created only a MessageStream. Indeed, the test application will be labeled jar Consumer packages, and passing different parameters in four different operating command line terminal.

1.2 High Level Consumer Rebalance

　　This section describes the contents are based on the relevant Rebalance High Level Consumer Kafka used to live)
　　Kafka used to live in the same guarantee only one Consumer Consumer Group will be spending a message, in fact, Kafka ensure that every instance of a Consumer steady state will only consume one or more Partition of a specific data, and a Partition data will only be consumed by a particular consumer instances. That Kafka distribution of the message is Partition units assigned to each message, rather than as a distribution unit. The disadvantage of this design is no guarantee that the same Consumer Group in the Consumer uniform consumption data, the advantage is not related to each Consumer lot of Broker communication, reduce communication costs, but also reduces the difficulty of distribution, implementation is easier. In addition, because the data in the same Partition is ordered, this design ensures that each Partition where the data can be ordered consumption.
　　If a Consumer Group in Consumer (per Consumer 1 Ge MessageStream create only) fewer than the number of Partition, then at least a Consumer will be spending more Partition of data, if the number of Consumer and same number of Partition, a Consumer consumption is just a Partition data. And if the number is greater than the number of Consumer Partition, there will be a message to any part of the Consumer under the Topic can not consume.
　　The following example, if a total of three Partition 0,1,2 topic1 has, when only a Consumer group1 (named consumer1), the Consumer consumable Partition all three of the data.

　　After adding a Consumer (consumer2), wherein a Consumer (consumer1) consumable 2 Partition the data (and Partition 0 Partition 1), a further Consumer (consumer2) additionally a consumable Partition (Partition 2) data.

After a further addition of a Consumer (consumer3), each of the consumable Consumer data of a Partition. consumer1 consumption partition0, consumer2 consumption partition1, consumer3 consumption partition2.

After a further addition of a Consumer (consumer4), of which three are a consumable Consumer Partition data, another Consumer (consumer4) does not consume any of topic1 data.

Now closed consumer1, the remaining three Consumer a consumable Partition data, respectively.

Then closed consumer2, consumer3 consumable 2 Partition, consumer4 consumable a Partition.

Then close consumer3, the only consumer4 simultaneous consumption of three Partition topic1.

　Consumer Rebalance algorithm is as follows:

All Partirtion sort the target Topic, stored in $P_{T}$
All Consumer ordering is a Consumer Group, stored $于 C_{G}$
$N = s i z e (P_{T}) / s i z e (C_{G})$
Lift $C_{i}$
The first $i * N$

At present, the control strategy Consumer Rebalance the latest version (0.8.2.1) Kafka is a Consumer pass by each register on the Zookeeper Watch completed. Each Consumer Consumer Group will trigger the Rebalance when they are created, specific startup process is as follows:

When the High Level Consumer startup register its ID to which the Consumer Group, on the path of Zookeeper/consumers/[consumer group]/ids/[consumer id]
In /consumers/[consumer group]/idsthe registration Watch
In /brokers/idsthe registration Watch
If the Consumer Creating a message flow by Topic Filter, at the same time it will /brokers/topicsalso create a Watch on
Rebalance force yourself to start the process within its Consumer Group

　　In this strategy, each Consumer Broker or increase or decrease will trigger Consumer Rebalance. Because each Consumer is responsible only for their own consumption to adjust the Partition, the entire Consumer Group in order to ensure consistency, when a Consumer triggered Rebalance, all other other Consumer in the Consumer Group should simultaneously trigger Rebalance.

　　This embodiment has the following drawbacks:

Herd effect
　　of any increase or decrease in the Consumer Broker or Rebalance will trigger all of the Consumer
Split Brain
　　Each Broker Consumer and Consumer downtime which were judged by a separate Zookeeper, then different Consumer at the same time from Zookeeper "look" to the View may be different, which is determined by the characteristics of the Zookeeper , which will result in incorrect Reblance try.
Adjusted results uncontrollable
　　All Consumer does not know whether all the other Consumer Rebalance successful, this could lead to Kafka an incorrect state work .

　　According to Kafka community wiki, Kafka authors are considering has not been released using the center Coordinator (Coordinator) 0.9.x versions . The general idea is that a subset of all Broker is elected as the Consumer Group Coordinator, by its Watch Zookeeper, to determine whether there is a Consumer Partition or decrease, then generates Rebalance command, and check if these are all related Rebalance in Consumer is successful, if unsuccessful retry, if successful, is considered the Rebalance success (this process is very similar with Replication Controller). In specific embodiments will be described in detail later.

0x02 Low Level Consumer

Mainly due to the use of Low Level Consumer (Simple Consumer) it is that users want to consume more than the Consumer Group of control data. such as:

Read the same message multiple times
Read only a portion of Partition Topic
Management, to ensure that each message is processed once and only once to be processed

　　Compared with the Consumer Group, Low Level Consumer require the user to do a lot of extra work.

Must track offset in the application, so a consumer under which message should determine
Who application needs to know each Partition by the program is Leader
Leader of the change must be addressed

　　Low Level Consumer using the following general procedure

Find a "alive" Broker, and to find the Leader of each Partition
Find Follower of each Partition
Defined the request, the request should be able to describe what data the application needs
Fetch data
Leader of the change identification, and make the necessary response

0x03 Consumer redesign

　　According to Community wiki, Kafka 0.9. * Version, redesigned Consumer is probably one of the most important Feature. This section will Kafka 0.9. * In a possible design direction for Consumer and ideas based on introduction community wiki.

3.1 design direction

Simplify the consumer client
　　some users want the development and use of non-java client. Stage using non-java hair SimpleConsumer more convenient, but want to develop a High Level Consumer is not easy. Because the High Level Consumer need to implement some complex but essential failure detection and Rebalance. If consumers can more streamlined client, minimizing the dependence will realize their Consumer greatly facilitate non-java user.
　　
Center Coordinator
　　As described above, the presence Herd Effect Split Brain and problems of the current version of High Level Consumer. If you fail to detect and Rebalance of logic into a high-availability center Coordinator, then these two issues can be solved. But can also greatly reduce the load Zookeeper, it is conducive to the Scale Out Kafka Broker.
　　
Allow manual management offset
　　some of the desired system at specific time intervals in a custom Offset management database. This requires the Consumer can obtain the metadata for each message, e.g. Topic, Partition, Offset, Offset also need to be activated each time Consumer Partition. Achieve these, we need to provide new Consumer API. At the same time there is a problem had to be considered, namely whether to allow manual management section Topic of Consumer Offset, and let Kafka automatically by Offset Zookeeper management of other Topic. One possible option is to have each Consumer can only select one kind Offset management mechanism, which can greatly simplify the design and implementation of the Consumer API.
　　
Rebalance triggered after a user-specified callback
　　Some applications might maintain some states for each Partition in memory, Rebalance, they may need to be persistent state. Therefore we wish to support the needs of users to achieve and specify the number of callbacks and triggered Rebalance when pluggable. If the user manually Offset management, demand that the user can easily get implemented, and if the user wishes to use the automatic Offset Management Kafka offer, you need to Kafka provides the callback mechanism.

Consumer API nonblocking
　　the demand from those achieved high flow processing operations, such as filter by, group by, join, etc., systems. Consumer stage blocking almost impossible Join operation.

How to achieve Rebalance ## through the center Coordinator
　　successful outcome Rebalance is subscribed all Topic of each Partition will be in a Consumer Group (and only one) Consumer ownership. Each Broker will be elected as Coordinator of certain Consumer Group. A Cosnumer Group's Coordinator is responsible for coordinating operations in the Partititon Rebalance change Topic changes or members of the Consumer Group subscribed.

Consumer
　　. 1) starts when the Consumer, Xianxiang any Broker Broker transmitting a list ConsumerMetadataRequest, and it acquires information Group Coordinator by ConsumerMetadataResponse. And has the following structure ConsumerMetadataRequest ConsumerMetadataResponse

ConsumerMetadataRequest
{
  GroupId                => String
}

ConsumerMetadataResponse
{
  ErrorCode              => int16
  Coordinator            => Broker
}

　　2) Consumer and connected to the transmission Coordinator HeartbeatRequest, if no error is returned HeartbeatResponse code, Consumer continue to fetch data. If IllegalGeneration which contains the error code that illustrates Coordinator has initiated Rebalance operations, this time Consumer stop fetch data, commit offset, and send JoinGroupRequest to its Coordinator, and get all the Partition it should have in JoinGroupResponse list and it belongs Group new Generation ID. At this point Rebalance complete, Consumer began to fetch data. Request and Response corresponding to the following structure

HeartbeatRequest
{
  GroupId                => String
  GroupGenerationId      => int32
  ConsumerId             => String
}

HeartbeatResponse
{
  ErrorCode              => int16
}

JoinGroupRequest
{
  GroupId                     => String
  SessionTimeout              => int32
  Topics                      => [String]
  ConsumerId                  => String
  PartitionAssignmentStrategy => String
}

JoinGroupResponse
{
  ErrorCode              => int16
  GroupGenerationId      => int32
  ConsumerId             => String
  PartitionsToOwn        => [TopicName [Partition]]
}
TopicName => String
Partition => int32

Consumer state machine

　　Down: Consumer stop working
　　Start up & discover coordinator: Consumer detect its Coordinator of the Group. Once it detects Coordinator, i.e. send JoinGroupRequest.
　　Part of a group: In this state, is already a member of the Consumer Group, and periodically transmits HeartbeatRequest. The HeartbeatResponse comprising IllegalGeneration error code, the state transitions to Stopped Consumption. If the connection is lost, HeartbeatResponse comprising NotCoordinatorForGroup error code, the state transitions to Rediscover coordinator.
　　Rediscover coordinator: In this state, Consumer consumption but does not stop attempts by sending ConsumerMetadataRequest to detect new Coordinator, and wait until the error code response.
　　Stopped consumption: In this state, Consumer stop spending and submitted offset, until it joins Group again.

Fault detection mechanism
　　after successfully joined the Consumer Group, Consumer Coordinator started simultaneously and the corresponding fault detection procedure. Consumer initiated periodic Heartbeat (HeartbeatRequest) and wait for response to the Coordinator, the period of session.timeout.ms/heartbeat.frequency. If the Consumer in session.timeout.ms not received HeartbeatResponse, or find the appropriate Socket channel disconnected, it assumes that the Coordinator is down and started Coordinator detection program. If the Coordinator session.timeout.ms not received within a HeartbeatRequest, marking it as the Consumer down state where its Group and trigger a Rebalance operation.
　　Coordinator Failover process, Consumer may be completed in the new Coordinator discover new Coordinator and send HeatbeatRequest before or after Failover process. For the latter, the new Cooodinator may reject the request, causing the Consumer reprobed Coordinator and initiate a new connection request. If the Consumer sends a new connection request Coordinator late, new Coordinator may have before it is marked down state of the newly joined and Consumer considered a trigger Rebalance operation.

Coordinator
　　1) Under steady state, by the above fault detection mechanisms Coordinator health status of each track at each Consumer Group manages.
　　2) After the election just started or completed, Coordinator Zookeeper read it manages from a list of members of the Group and the Group of the list. If you do not get to the Group membership information, it will not do anything until some Group members have registered to come.
　　Before 3) complete the list of Group and their respective members of its management information loaded in the Coordinator, it returns an error code CoordinatorStartupNotComplete HeartbeatRequest, OffsetCommitRequest and JoinGroupRequests. At this time, Consumer will resend requests.
　　4) Coordinator will track changes in Partition is registered under their administration of any Consumer Group Topic and triggers Rebalance operations for change. Create a new Topic may also trigger Rebalance, because the Consumer can already subscribe to it before the Topic is created.
　　Coordinator Rebalance initiates operation flow shown below.

Coordinator state machine

　 Down: Responsible parties no longer serve as Coordinator of the Consumer Group Coordinator
　　Catch up: this state, Coordinator win the election, but are not ready to service the appropriate request.
　　Ready: In this state, the new campaign has finished loading out of the Coordinator is responsible for its management of metadata from all Group Zookeeper, and can start receiving the corresponding request.
　　Prepare for rebalance: In this state, Coordinator IllegalGeneration returns an error code in the all HeartbeatResponse, and waits for the transmission to which all Consumer JoinGroupRequest to Rebalancing state.
　　Rebalancing: In this state, JoinGroupRequest Coordinator has received the request, and to increase the Group Generation ID, assigned Consumer ID, assigned Partition. After Rebalance successful, it waits for receiving a new HeartbeatRequest Consumer Generation ID's, and go to the Ready state.

Coordinator Failover
　　As previously mentioned, Rebalance operations need to go through the following phases
　　1) Topic Partition changes / or the addition of new or existing Consumer Consumer stop, triggering Coordinator Zookeeper register on the watch, Coordinator notified preparing to launch operations Rebalance .
　　2) Coordinator IllegalGeneration by returning an error code to initiate operation HeartbeatResponse in Rebalance.
　　3) Consumer transmission JoinGroupRequest
　　. 4) Generation ID Group Coordinator increase in Zookeeper and writes the new Partition distribution Zookeeper
　　. 5) transmits Coordinator JoinGroupResponse
　　
　　each stage in this process, Coordinator, errors may occur. Failover given below Rebalance in different stages of treatment of the Coordinator.
　　1) If the Coordinator of failures in the first stage, that it has received Notification does not have time to respond, the new Coordinator Zookeeper will read metadata Group, including the Group of Topic subscription list and Partition assignment before. If the Partition number Partition and assignment before a number Topic Group subscribed to a Topic or inconsistent, or by connecting to a Group Consumer several new Coordinator of inconsistent Partition allocation before the new Coordinator will initiate Rebalance operating.
　　2) If a failure occurs in Phase 2, it may issue HeartbeatResponse with an error code on some, but not all of the Consumer. As with the first case of the above, the new Coordinator will detect and initiate a necessity Rebalance Rebalance operations. If Rebalance is triggered by the failure of Consumer and Cosnumer before the Coordinator of Failover recovery is completed, the new Coordinator for this purpose will not initiate a new Rebalance operations.
　　3) If a Failure occurs in phase 3, the new Coordinator may receive only some but not all of Consumer JoinGroupRequest. After Failover is complete, it may receive JoinGroupRequest part of HeartRequest Consumer and another part of the Consumer. Similar to the first case, it will launch a new round of Rebalance operations.
　　4) Failure occurred if at stage 4, i.e. it is a new member of Group Group Generation ID and the information writing Zookeeper. New Generation ID and Group membership information is written to an atomic operation Zookeeper disposable. After Failover completed, Consumer HeartbeatRequest will be sent to the new Coordinator, and contain old Generation ID. At this time, the new Coordinator IllegalGeneration by returning an error code to launch a new round Rebalance in HeartbeatResponse in. This also explains why every HeartbeatRequest both need to include Generation ID and the Consumer ID.
　　5) Failure occurred if at stage 5, the old JoinGroupResponse Coordinator may send only part of the Group Consumer. It detects that it has failed to receive JoinGroupResponse the Consumer has failed to send HeartbeatRequest Coordinator at the next or submit Offset. At this time, it detects the new Coordinator and send HeartbeatRequest with the new Generation ID. JoinGroupResponse not received the Consumer will detect the new Coordinator and send JoinGroupRequest, which will lead to a new Coordinator launch a new round of Rebalance.