ConsumerRebalanceListener
Rebalancing the listener to set some preparation before and after the occurrence of re-balancing action or ending action. ConsumerRebalanceListener is an interface includes two methods:
void onPartitionsRevoked(Collectionpartitions)
This method will be called after the equalizer before the start and stop reading the news consumer. Consumer bits can be processed by this method
Shift submission, in order to avoid unnecessary duplication of spending phenomenon.
Parameter indicates partitions before equalization and then assigned to the partition
void onPartitionsAssigned(Collectionpartitions)
This method will be called before consumers start reading and consumption after redistribution. After re-balancing parameters partitions represent the points
Assigned to the partition.
Simple to use listener
MapcurrentOffsets = new HashMap();
consumer.subscribe(Arrays.asList(topic), new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collectionpartitions) {
// submit consumer displacement
consumer.commitSync(currentOffsets);
}
@Override
public void onPartitionsAssigned(Collectionpartitions) {
//do nothing.
}
});
try {
while (true) {
ConsumerRecordsrecords =
consumer.poll(Duration.ofMillis(100));
for (ConsumerRecordrecord : records) {
// Consumer news
currentOffsets.put(
new TopicPartition(record.topic(), record.partition()),
new OffsetAndMetadata(record.offset() + 1));
}
consumer.commitAsync(currentOffsets, null);
}
} finally {
consumer.close();
}
Use seek the
General equilibrium again when you can fit seek methods prescribed displacements consumption.
KafkaConsumer provides seek () method:
public void seek(TopicPartition partition,long offset)
seek () method partition represents the partition, and offset parameters are used to specify the starting position from which the consumer partition.
seek () method can only be reset consumer consumption position assigned to the partition, a partition is assigned during the call poll () method
Achieved.
Consumption can reset the bit after that is to say you need to run a poll before executing seek () method () method until assigned to Area
Home.
KafkaConsumerconsumer = new KafkaConsumer(props);
consumer.subscribe(Arrays.asList(topic));
long start = System.currentTimeMillis();
Setassignment = new HashSet ();
while (assignment.size() == 0) {
consumer.poll(Duration.ofMillis(100));
// Get the information consumer is assigned to partition
assignment = consumer.assignment();
}
long end = System.currentTimeMillis();
System.out.println(end - start);
System.out.println(assignment);
for (TopicPartition tp : assignment) {
consumer.seek(tp, 10);
}
Therefore rebalancing occurs when we can use in onPartitionsAssigned seek to acquire start spending displacement, be
Set up
Use of the flume
In the flume in KafkaSource also been re-balancing process.
doStart ():
// Subscribe for topics by already specified strategy
subscriber.subscribe(consumer, new SourceRebalanceListener(rebalanceFlag));
Specific implementation is SourceRebalanceListener:
class SourceRebalanceListener implements ConsumerRebalanceListener {
private static final Logger log = LoggerFactory.getLogger(SourceRebalanceListener.class);
private AtomicBoolean rebalanceFlag;
public SourceRebalanceListener(AtomicBoolean rebalanceFlag) {
this.rebalanceFlag = rebalanceFlag;
}
// Set a flag that a rebalance has occurred. Then commit already read events to kafka.
public void onPartitionsRevoked(Collectionpartitions) {
for (TopicPartition partition : partitions) {
log.info(topic {} - partition {} revoked., partition.topic(), partition.partition());
rebalanceFlag.set(true);
}
}
public void onPartitionsAssigned(Collectionpartitions) {
for (TopicPartition partition : partitions) {
log.info(topic {} - partition {} assigned., partition.topic(), partition.partition());
}
}
}
It can be seen in the flume, if the equalization has occurred and then executes in two corresponding logical callback methods mentioned above.
The specific content of execution you can read the source code to learn.
Follow-up
Spark Streaming+kafka
Reproduced in: https: //juejin.im/post/5cef85326fb9a07f014ed96d