kafka use in java

 

kafka  Producer  Api

Procuder API, there are two: kafka.producer.SyncProducer and kafka.producer.async.AsyncProducer they have achieved the same interface:

class Producer {
　　/ * the message to a specified partition * / 
　　publicvoid Send (kafka.javaapi.producer.ProducerData <K, V> producerData);
 
　　/ * Batch number of transmitted message * / 
　　publicvoid Send (java.util.List <Kafka .javaapi.producer.ProducerData <K, V >> producerData);
 
　　/ * Close Producer * / 
　　publicvoid Close (); 
}

 

Producer API provides the following functions:

• A plurality of message may be cached in the local queue, and then sent to the broker asynchronous batch can be done by parameter = async producer.type. Cache size can be specified by a number of parameters: queue.time and batch.size. A background thread ((kafka.producer.async.ProducerSendThread) data extracted from the queue and sends a message to let kafka.producer.EventHandler broker, it can also be customized parameter handler event.handler, at different stages of processing data producer side Sign processors, such as the process can be carried out on a log to track, monitor or some. just realize kafka.producer.async.CallbackHandler interface and configuration callback.handler in.
• Encoder to write your own message serialization, simply implement the following interfaces. The default Encoder is kafka.serializer.DefaultEncoder.
  
  • interface Encoder<T> {
  • public Message toMessage(T data);
  • }
• It provides the ability to automatically sense based broker Zookeeper can be achieved by means of parameters zk.connect. If you do not use Zookeeper, you can use broker.list parameter specifies a static list of brokers, this message will be randomly sent to a broker, the broker failed once selected, the message sent will fail.
• Partitioned by the partition function kafka.producer.Partitioner message class.
  
  • interface Partitioner<T> {
  • int partition(T key, int numPartitions);
  • }
  
  Partition function takes two parameters: key and the number of partitions available, select a partition from the partition and returns the list id. The default partitioning strategy is to hash (key)% numPartitions. If the key is null, it is a random choice. You can customize the partition function parameter partitioner.class.

 

import java.util.*;

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;


public class ASyncProduce {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put("metadata.broker.list", "192.168.1.1:19092,192.168.1.2:19092,192.168.1.3:19092");
        props.put("serializer.class", "kafka.serializer.StringEncoder");//kafka.serializer.DefaultEncoder
        props.put("partitioner.class", "kafka.producer.partiton.SimplePartitioner");
        //kafka.producer.DefaultPartitioner: based on the hash of the key
        //props.put("request.required.acks", "1");
        props.put("producer.type", "async");//1: async 2: sync
        //props.put("producer.type", "1");
        // 1: async 2: sync
 
        ProducerConfig config = new ProducerConfig(props);
 
        Producer<String, String> producer = new Producer<String, String>(config);
 
        for (int i = 0; i < Integer.MAX_VALUE; i++) { 
               long runtime = new Date().getTime();  
               String ip = "192.168.2.1"; 
               String msg = "message"; 
               KeyedMessage<String, String> data = new KeyedMessage<String, String>("topic", ip, msg);
               producer.send(data);
               try {
                   Thread.sleep(1000);
               } catch (InterruptedException e) {
                   e.printStackTrace();
               }
        }
        producer.close();
    }
}

 

import kafka.javaapi.producer.Producer;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;

public class SyncProduce {
    public static void main(String[] args) {
 
        Properties props = new Properties();
        props.put("metadata.broker.list", "192.168.1.1:19092,192.168.1.2:19092,192.168.1.3:19092");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        //kafka.serializer.DefaultEncoder
        props.put ( "partitioner.class", "kafka.producer.partiton.SimplePartitioner" );
         // kafka.producer.DefaultPartitioner: The ON based The Key of the hash 
        props.put ( "request.required.acks", ". 1 " );
         // 0; 1 never confirmed such as: a copy of the leader of the receipt of this message, and sends back an acknowledgment -1: all have received a copy of the leader of this message, and sends back an acknowledgment 
 
        ProducerConfig config = new new ProducerConfig (props ); 
 
        Producer <String, String> Producer = new new Producer <String, String> (config); 
 
        for ( int I = 0; I <Integer.MAX_VALUE; I ++ ) { 
               String IP = "192.168.2.1"; 
               Msg String = "the Message" ; 
                // eventKey must have (even if its own partition algorithm does not use this key, can not be set to null or ""), not those who own partition algorithms do not receive calls 
               KeyedMessage <String, String> Data = new new KeyedMessage <String, String> ( "Topic" , IP, MSG); 
               producer.send (Data); 
               the try { 
                   the Thread.sleep ( 1000 ); 
               } the catch (InterruptedException IE) { 
               } 
        } 
        producer.close ( ); 
    } 
}

 

 

Consumer APIs Kafka used to live

Consumer API There are two levels. And a low-level specified broker remain connected, and close the connection after receiving the message, this level is stateless, with each read messages are offset.
High-level API hides the details and brokers connections, without having to be concerned about server architecture and server communication. They can also maintain their own consumption state, and subscribe to a particular topic can be specified through a number of conditions, such as a whitelist or blacklist regular expressions.

 

 

import kafka.api.FetchRequest;
import kafka.api.FetchRequestBuilder;
import kafka.api.PartitionOffsetRequestInfo;
import kafka.common.ErrorMapping;
import kafka.common.TopicAndPartition;
import kafka.javaapi.*;
import kafka.javaapi.consumer.SimpleConsumer;
import kafka.message.MessageAndOffset;
 
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class PartitionConsumerTest {
    
    public static void main(String args[]) {
        PartitionConsumerTest example = new PartitionConsumerTest();
        long maxReads = Long.MAX_VALUE;
        String topic = "topic";
        if(args.length < 1){
            System.out.println("Please assign partition number.");
        }
        
        List<String> seeds = new ArrayList<String>();
        seeds.add("192.168.2.1");
        seeds.add("192.168.2.2");
        seeds.add("192.168.2.3");
        
        int port = 19092;
         
        int partLen = Integer.parseInt(args[0]);
        for(int index=0;index < partLen;index++){
            try {
                example.run(maxReads, topic, index/*partition*/, seeds, port);
            } catch (Exception e) {
                System.out.println("Oops:" + e);
                 e.printStackTrace();
            }
        }
    }
    
    private List<String> m_replicaBrokers = new ArrayList<String>();
     
    public PartitionConsumerTest() {
        m_replicaBrokers = new ArrayList<String>();
    }
 
    public void run(long a_maxReads, String a_topic, int a_partition, List<String> a_seedBrokers, int a_port) throws Exception {

        PartitionMetadata metadata = findLeader(a_seedBrokers, a_port, a_topic, a_partition);
        if (metadata == null) {
            System.out.println("Can't find metadata for Topic and Partition. Exiting");
            return;
        }
        if (metadata.leader() == null) {
            System.out.println("Can't find Leader for Topic and Partition. Exiting");
            return;
        }
        String leadBroker = metadata.leader().host();
        String clientName = "Client_" + a_topic + "_" + a_partition;
 
        SimpleConsumer consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
        long readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.EarliestTime(), clientName);
 
        int numErrors = 0;
        while (a_maxReads > 0) {
            if (consumer == null) {
                consumer = new SimpleConsumer(leadBroker, a_port, 100000, 64 * 1024, clientName);
            }
            FetchRequest req = new FetchRequestBuilder()
                    .clientId(clientName)
                    .addFetch(a_topic, a_partition, readOffset, 100000) // Note: this fetchSize of 100000 might need to be increased if large batches are written to Kafka
                    .build();
            FetchResponse fetchResponse = consumer.fetch(req);
 
            if (fetchResponse.hasError()) {
                numErrors++;
                // Something went wrong!
                short code = fetchResponse.errorCode(a_topic, a_partition);
                System.out.println("Error fetching data from the Broker:" + leadBroker + " Reason: " + code);
                if (numErrors > 5) break;
                if (code == ErrorMapping.OffsetOutOfRangeCode())  {
                    // We asked for an invalid offset. For simple case ask for the last element to reset
                    readOffset = getLastOffset(consumer,a_topic, a_partition, kafka.api.OffsetRequest.LatestTime(), clientName);
                    continue;
                }
                consumer.close();
                consumer = null;
                leadBroker = findNewLeader(leadBroker, a_topic, a_partition, a_port);
                continue;
            }
            numErrors = 0;
 
            long numRead = 0;
            for (MessageAndOffset messageAndOffset : fetchResponse.messageSet(a_topic, a_partition)) {
                long currentOffset = messageAndOffset.offset();
                if (currentOffset < readOffset) {
                    System.out.println("Found an old offset: " + currentOffset + " Expecting: " + readOffset);
                    continue;
                }
                readOffset = messageAndOffset.nextOffset();
                ByteBuffer payload = messageAndOffset.message().payload();
 
                byte[] bytes = new byte[payload.limit()];
                payload.get(bytes);
                System.out.println(String.valueOf(messageAndOffset.offset()) + ": " + new String(bytes, "UTF-8"));
                numRead++;
                a_maxReads--;
            }
 
            if (numRead == 0) {
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException ie) {
                }
            }
        }
        if (consumer != null) consumer.close();
    }
 
    public static long getLastOffset(SimpleConsumer consumer, String topic, int partition,
                                     long whichTime, String clientName) {
        TopicAndPartition topicAndPartition = new TopicAndPartition(topic, partition);
        Map<TopicAndPartition, PartitionOffsetRequestInfo> requestInfo = new HashMap<TopicAndPartition, PartitionOffsetRequestInfo>();
        requestInfo.put(topicAndPartition, new PartitionOffsetRequestInfo(whichTime, 1));
        kafka.javaapi.OffsetRequest request = new kafka.javaapi.OffsetRequest(
                requestInfo, kafka.api.OffsetRequest.CurrentVersion(), clientName);
        OffsetResponse response = consumer.getOffsetsBefore(request);
 
        if (response.hasError()) {
            System.out.println("Error fetching data Offset Data the Broker. Reason: " + response.errorCode(topic, partition) );
            return 0;
        }
        long[] offsets = response.offsets(topic, partition);
        return offsets[0];
    }
 
    private String findNewLeader(String a_oldLeader, String a_topic, int a_partition, int a_port) throws Exception {
        for (int i = 0; i < 3; i++) {
            boolean goToSleep = false;
            PartitionMetadata metadata = findLeader(m_replicaBrokers, a_port, a_topic, a_partition);
            if (metadata == null) {
                goToSleep = true;
            } else if (metadata.leader() == null) {
                goToSleep = true;
            } else if (a_oldLeader.equalsIgnoreCase(metadata.leader().host()) && i == 0) {
                // first time through if the leader hasn't changed give ZooKeeper a second to recover
                // second time, assume the broker did recover before failover, or it was a non-Broker issue
                //
                goToSleep = true;
            } else {
                return metadata.leader().host();
            }
            if (goToSleep) {
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException ie) {
                }
            }
        }
        System.out.println("Unable to find new leader after Broker failure. Exiting");
        throw new Exception("Unable to find new leader after Broker failure. Exiting");
    }
 
    private PartitionMetadata findLeader(List<String> a_seedBrokers, int a_port, String a_topic, int a_partition) {
        PartitionMetadata returnMetaData = null;
        loop:
        for (String seed : a_seedBrokers) {
            SimpleConsumer consumer = null;
            try {
                consumer = new SimpleConsumer(seed, a_port, 100000, 64 * 1024, "leaderLookup");
                List<String> topics = Collections.singletonList(a_topic);
                TopicMetadataRequest req = new TopicMetadataRequest(topics);
                kafka.javaapi.TopicMetadataResponse resp = consumer.send(req);
 
                List<TopicMetadata> metaData = resp.topicsMetadata();
                for (TopicMetadata item : metaData) {
                    for (PartitionMetadata part : item.partitionsMetadata()) {
                        if (part.partitionId() == a_partition) {
                            returnMetaData = part;
                            break loop;
                        }
                    }
                }
            } catch (Exception e) {
                System.out.println("Error communicating with Broker [" + seed + "] to find Leader for [" + a_topic
                        + ", " + a_partition + "] Reason: " + e);
            } finally {
                if (consumer != null) consumer.close();
            }
        }
        if (returnMetaData != null) {
            m_replicaBrokers.clear();
            for (kafka.cluster.Broker replica : returnMetaData.replicas()) {
                m_replicaBrokers.add(replica.host());
            }
        }
        return returnMetaData;
    }
}

 

 

import kafka.consumer.ConsumerConfig;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
 
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class GroupConsumerTest extends Thread {
    private final ConsumerConnector consumer;
    private final String topic;
    private  ExecutorService executor;
    
    public GroupConsumerTest(String zookeeper, String groupId, String topic){
        consumer = kafka.consumer.Consumer.createJavaConsumerConnector(createConsumerConfig(zookeeper, groupId));
        this.topic = topic;
    }
    
    public void shutdown() {
        if (consumer != null) consumer.shutdown();
        if (executor != null) executor.shutdown();
        try {
            if (!executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS)) {
                System.out.println("Timed out waiting for consumer threads to shut down, exiting uncleanly");
            }
        } catch (InterruptedException e) {
            System.out.println("Interrupted during shutdown, exiting uncleanly");
        }
   }
 
    public void run(int a_numThreads) {
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(a_numThreads));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);
 
        executor = Executors.newFixedThreadPool(a_numThreads);
 
        int threadNumber = 0;
        for (final KafkaStream stream : streams) {
            executor.submit(new ConsumerTest(stream, threadNumber));
            threadNumber++;
        }
    }
    
    private static ConsumerConfig createConsumerConfig(String zookeeper, StringgroupId) {
        Properties props = new Properties();
        props.put("zookeeper.connect", zookeeper);
        props.put("group.id", groupId);
        props.put("zookeeper.session.timeout.ms", "40000");
        props.put("zookeeper.sync.time.ms", "2000");
        props.put("auto.commit.interval.ms", "1000");
        return new ConsumerConfig(props);
    }
    
    public static void main(String[] args) {
        if(args.length < 1){
            System.out.println("Please assign partition number.");
        }
        
        String zooKeeper = "192.168.1.1:12181,192.168.1.1:12181,192.168.1.1:12181";
        String groupId = "grouptest";
        String topic = "topic";
        int threads = Integer.parseInt(args[0]);
 
        GroupConsumerTest example = new GroupConsumerTest(zooKeeper, groupId, topic);
        example.run(threads);
 
        try {
            Thread.sleep(Long.MAX_VALUE);
        } catch (InterruptedException ie) {
 
        }
        example.shutdown();
    }
}

 

 

 

Reference: https: //blog.csdn.net/tangdong3415/article/details/53432166