Kafka producer custom partitioning strategy

Implement a partitioning strategy for the audit function: Suppose we have two kinds of messages, in which the key message is a kind of audit, for auditing, on the last partition, the other messages randomly assigned in the rest of the partition.

Three partitions to create a copy of the three themes audit-test:

bin/kafka-topics.sh --create --zookeeper localhost:2181,localhost:2812,localhost:2183 --topic audit-test 
--partitions 3 --replication-factor 3
//查看
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic audit-test

Kafka then implement Partitioner interfaces provided by the client:

Import org.apache.kafka.clients.producer.Partitioner;
 Import org.apache.kafka.common.Cluster;
 Import org.apache.kafka.common.PartitionInfo; 

Import java.util.List;
 Import a java.util.Map;
 Import java.util.Random; 


/ ** * 
 * 
 * implement a custom partitioning strategy: 
 * 
 * Key portion comprises sending a message to the audit partition on the last, other messages are randomly assigned to other partitions 
 * 
 * 
 * 
 * / 
public  class PartitionerImpl the implements {Partitioner 


    Private Random Random; 

    public  void the configure (the Map <? String,> configs) {
        //做必要的初始化工作
        random = new Random();
    }

    //分区策略
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {

        String keyObj = (String) key;
        List<PartitionInfo> partitionInfoList = cluster.availablePartitionsForTopic(topic);
        int partitionCount = partitionInfoList.size();
        System.out.println("partition size: "  + partitionCount);
        int auditPartition = partitionCount - 1;
        return keyObj == null || "".equals(keyObj) || !keyObj.contains("audit") ? random.nextInt(partitionCount - 1) : auditPartition;
    }

    public void close() {
        //清理工作
    }
}

Then set the startup class parameters:

//实现类
properties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,"cn.org.fubin.PartitionerImpl");

String topic = "audit-test";
Producer<String,String> producer = new KafkaProducer<String, String>(properties);
ProducerRecord nonKeyRecord = new ProducerRecord(topic,"non-key record");
//这类消息需要放在最后一个分区
ProducerRecord auditRecord = new ProducerRecord(topic,"audit","audit record");
ProducerRecord nonAuditRecord = new ProducerRecord(topic,"other","non-audit record");

try {
    producer.send(nonAuditRecord).get();
    producer.send(nonAuditRecord).get();
    producer.send(auditRecord).get();
    producer.send(nonAuditRecord).get();
    producer.send(nonAuditRecord).get();
} catch (InterruptedException e) {
    e.printStackTrace();
} catch (ExecutionException e) {
    e.printStackTrace();
}

Last Verification: more than a few times push messages, view the number of messages each partition

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic audit-test

Kafka producer custom partitioning strategy

Guess you like