Kafka- independent consumer

Kafka- independent consumer

Sometimes, we just need a consumer to read data from all partitions or a partition of a particular topic. Then you do not need consumer groups and re-balanced, and just need to themes or partition to the consumer, then start reading the message and submit the offset.

If this is the case, I do not subscribe to topics, replaced by a partition allocated for himself.

code show as below

import com.chinaventure.kafka.serializer.Customer;
import com.chinaventure.util.ExceptionUtil;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.PartitionInfo;
import org.apache.kafka.common.TopicPartition;

import java.time.Duration;
import java.util.*;

/**
 * @Author FengZhen
 * @Date 2020-04-06 11:07
 * @Description kafka消费者
 */
public class KafkaConsumerTest {
    private static Properties kafkaProps = new Properties();
    static {
        kafkaProps.put("bootstrap.servers", "localhost:9092");
        kafkaProps.put("group.id", "test");
        kafkaProps.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        kafkaProps.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    }

    private staticMap <TopicPartition, OffsetAndMetadata> currentOffsets = new new the HashMap <> (); 

    Private  static KafkaConsumer <String, String> Consumer; 

    public  static  void main (String [] args) {
         closeSoftly ();
         singleConsumer (); 
    } 

    / * * 
     * independently consumers 
     * consumers can assign their own partition, do not need to subscribe to topics, rebalancing does not happen, there is no concept of the group 
     * drawbacks: If the subject of the new partition, consumers will not receive notification, so either periodically call consumer.partitionsFor () method to check whether a new partition is added, either after the addition of a new partition to restart the application 
     * / 
    public  static  void singleConsumer () { 
        Consumer = new new KafkaConsumer <String, String>(kafkaProps);
        List<PartitionInfo> partitionInfos = consumer.partitionsFor("test_partition");
        List<TopicPartition> partitions = new ArrayList<>();

        if (null != partitionInfos){
            for (PartitionInfo partitionInfo : partitionInfos) {
                partitions.add(new TopicPartition(partitionInfo.topic(), partitionInfo.partition()));
            }
            consumer.assign(partitions);
            try {
                while (true){
                    // consumers continue to be in rotation for kafka, otherwise they will be considered dead, it's partition will be handed over to the group in the other consumers.
                    // passed to the poll method is a timeout for blocking time control poll () methods (blocking happens when there is no data available in the buffer of the consumer)
                     // If this parameter is set to 0, poll will return immediately, otherwise it will wait for the specified number of milliseconds broker returns data
                     // poll method returns a list of records. Each record contains information about the record belongs to the theme, the partition information is recorded, the record offset partition, as well as records of key pairs. 
                    ConsumerRecords <String, String> Records = consumer.poll (Duration.ofMillis ( 100 )); 
                    the System. OUT .println ( " ==== ==== GET Data " );
                     for (ConsumerRecord <String, String> Record: Records) { 
                        the System. OUT.println (String.format ( " Topic =% S, S% = Partition, offset =% D, Key =% S, S% = value " , 
                                record.topic (), record.partition (), record.offset ( ), record.key (), record.value ())); 
                    } 
                } 
            } the catch (Exception E) { 
                e.printStackTrace (); 
            } the finally {
                 // use close method Close before exiting consumer applications.
                // network connection and socket will also shut down, and immediately trigger a rebalancing, rather than waiting for the group coordinator found that it does not send a heartbeat and found it had died, because that takes longer, leading to political groups Brother Can not read the message at a time. 
                consumer.close (); 
            } 
        } 
    } 
/ ** 
     * Elegant exit 
     * consumer.wakeup () method is the only one who can secure consumers called from the other thread 
     * call this method, you can exit poll (), and throw WakeupException 
     * / 
    Private  static  void closeSoftly () { 
        Runtime. . getRuntime () addShutdownHook ( new new the Thread () { 
            @Override 
            public  void RUN () { 
                the System. OUT .println ( " Starting Exit ... " ); 
                consumer.wakeup (); 
            } 
        }); 
    } 
}