[Kafka] Kafka custom partitioner

1. The default partitioning strategy

(1) If the key is null, and a default partition is used, the record will be sent to the respective partitions within the subject matter available random. The partitioner polling (Round Robin) algorithm message equally distributed among the respective partition.
(2) if the key is not empty, and using the default partitioner, Kafka will then take the key and the hash value of the hash value mapping message to a particular partition. The key point here is that the same key is always mapped to the same partition, so during the mapping, we will use the theme of all the district, not just the available partitions. This also means that, if the write data partition is unavailable, an error will occur. But this rarely happens.

2. Custom partitioner

In order to meet business needs, you may want to customize the partition, for example, call history, call customer service record to be saved to a partition, the distribution of the rest of the record average of the remaining partition. We on this case to demonstrate.

(1) The partitioner Custom

package com.bonc.rdpe.kafka110.partitioner;

import java.util.List;
import java.util.Map;

import org.apache.kafka.clients.producer.Partitioner;
import org.apache.kafka.common.Cluster;
import org.apache.kafka.common.PartitionInfo;

/**
 * @Author: Stephen
 * @Date: 2020/2/25 15:18
 * @Content: 自定义分区器
 */
public class PhonenumPartitioner implements Partitioner{
    
    @Override
    public void configure(Map<String, ?> configs) {
        // TODO nothing
    }
    /**
     * 自定义kafka分区主要解决用户分区数据倾斜问题 提高并发效率（假设 3 分区）
     * @param topic 消息队列名
     * @param key 用户传入key
     * @param keyBytes key字节数组
     * @param value 用户传入value
     * @param valueBytes value字节数据
     * @param cluster 当前kafka节点数
     * @return 如果3个节点数 返回 0 1 2 如果5个 返回 0 1 2 3 4 5
     */
    @Override
    public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
        // 得到 topic 的 partitions 信息
        List<PartitionInfo> partitions = cluster.partitionsForTopic(topic);
        int numPartitions = partitions.size();
        // 模拟某客服
        if(key.toString().equals("10000") || key.toString().equals("11111")) {
            // 放到最后一个分区中
            return numPartitions - 1;
        }
        String phoneNum = key.toString();
        return phoneNum.substring(0, 3).hashCode() % (numPartitions - 1);
    }

    @Override
    public void close() {
        // TODO nothing
    }

}

(2) using a custom partitioner

package com.bonc.rdpe.kafka110.producer;

import java.util.Properties;
import java.util.Random;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

/**
 * @Title PartitionerProducer.java 
 * @Description 测试自定义分区器
 * @Author YangYunhe
 * @Date 2018-06-25 15:10:04
 */
public class PartitionerProducer {
    
    private static final String[] PHONE_NUMS = new String[]{
        "10000", "10000", "11111", "13700000003", "13700000004",
        "10000", "15500000006", "11111", "15500000008", 
        "17600000009", "10000", "17600000011" 
    };
    
    public static void main(String[] args) throws Exception {
        
        Properties props = new Properties();
        props.put("bootstrap.servers", "192.168.42.89:9092,192.168.42.89:9093,192.168.42.89:9094");
        // 设置分区器
        props.put("partitioner.class", "com.bonc.rdpe.kafka110.partitioner.PhonenumPartitioner");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        Producer<String, String> producer = new KafkaProducer<>(props);

        int count = 0;
        int length = PHONE_NUMS.length;
        
        while(count < 10) {
            Random rand = new Random();
            String phoneNum = PHONE_NUMS[rand.nextInt(length)];
            ProducerRecord<String, String> record = new ProducerRecord<>("dev3-yangyunhe-topic001", phoneNum, phoneNum);
            RecordMetadata metadata = producer.send(record).get();
            String result = "phonenum [" + record.value() + "] has been sent to partition " + metadata.partition();
            System.out.println(result);
            Thread.sleep(500);
            count++;
        }
        producer.close();
    }
}

(3) Test Results

phonenum [11111] has been sent to partition 2
phonenum [11111] has been sent to partition 2
phonenum [17600000009] has been sent to partition 0
phonenum [17600000011] has been sent to partition 0
phonenum [13700000003] has been sent to partition 1
phonenum [10000] has been sent to partition 2
phonenum [10000] has been sent to partition 2
phonenum [15500000008] has been sent to partition 1
phonenum [10000] has been sent to partition 2
phonenum [17600000009] has been sent to partition 0