Kafka--Interceptor-★★★

Introduction to interceptors

• Producer interceptor (interceptor) was introduced in Kafka 0.10 version, mainly used forRealize the customized control logic on the clients side.
• The interface of Intercetpor isorg.apache.kafka.clients.producer.ProducerInterceptor, Its definition methods include:
• 1.configure(configs)
  • Called when obtaining configuration information and initializing data.
• 2.onSend(ProducerRecord)
  • This method is encapsulated in the KafkaProducer.send method, that is, it runs in the user's main thread.Producer确保在消息被序列化以及计算分区前调用该方法。
  • The user can do any operation on the message in this method, but it is best to ensure not to modify the topic and partition to which the message belongs, otherwise it will affect the calculation of the target partition
• 3.onAcknowledgement(RecordMetadata, Exception)
  • 该方法会在消息被应答或消息发送失败时调用, And usually before the producer callback logic is triggered. onAcknowledgement runs in the producer's IO thread, so do not put heavy logic in this method, otherwise it will slow down the producer's message sending efficiency
• 4.close：
  • Close the interceptor,主要用于执行一些资源清理工作
  • As mentioned earlier, the interceptor may be run in multiple threads, so users need to ensure thread safety by themselves during specific implementation. In addition, if multiple interceptors are specified, the producer will call them in the specified order, and only capture the exceptions that may be thrown by each interceptor and record them in the error log instead of passing them up. This should be paid special attention during use.

demand analysis

• Implement a simple interception chain composed of dual interceptors.
  • The first interceptor codes the mobile phone number in the message, such as 13888888888--> 138****8888
  • The second interceptor counts the number of successfully sent messages or the number of failed messages after the message is sent

Code

Interceptor 1

package cn.hanjiaxiaozhi.interceptor;

import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Map;

/**
 * Author hanjiaxiaozhi
 * Date 2020/7/11 15:41
 * Desc 自定义拦截器1,实现将手机号中间4位打码,如13888888888--->138****8888,规定手机号在value中
 */
public class MyInterceptor1_Mosaic implements ProducerInterceptor<String,String> {
    
    
    //在真正发送到Kafka服务器之前调用
    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
    
    
        //获取原始手机号
        String phoneNum = record.value().toString();//13888888888
        //获取前3位+****+后4位
        String mosaicNum = phoneNum.substring(0, 3) + "****" + phoneNum.substring(7);
        return  new ProducerRecord(record.topic(),
                record.partition(),
                record.timestamp(),
                record.key(),
                mosaicNum);
    }

    //ack应答之后调用
    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
    
    

    }

    //拦截器执行完之后调用
    @Override
    public void close() {
    
    

    }

    //如果需要获取一些配置信息,可以在这获取
    @Override
    public void configure(Map<String, ?> configs) {
    
    

    }
}

Interceptor 2

package cn.hanjiaxiaozhi.interceptor;

import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Map;

/**
 * Author hanjiaxiaozhi
 * Date 2020/7/11 15:41
 * Desc 自定义拦截器2,实现发送数据统计
 */
public class MyInterceptor2_Statistics implements ProducerInterceptor<String,String> {
    
    
    private int successCount = 0;
    private int failCount = 0;
    //在真正发送到Kafka服务器之前调用
    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
    
    
        return record;
    }

    //ack应答之后调用
    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
    
    
        if(exception==null){
    
    
            //没有异常说明发送成功,成功次数+1
            successCount++;
        }else{
    
    
            //有异常说明发送失败,失败次数+1
            failCount++;
        }
    }

    //拦截器执行完之后调用
    @Override
    public void close() {
    
    
        //保存成功次数和失败次数/或者直接输出
        System.out.println("发送消息成功"+successCount+"次");
        System.out.println("发送消息失败"+failCount+"次");
    }

    //如果需要获取一些配置信息,可以在这获取
    @Override
    public void configure(Map<String, ?> configs) {
    
    

    }
}

Test-producer

package cn.hanjiaxiaozhi.producer;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;

import java.util.Arrays;
import java.util.Properties;

/**
 * Author hanjiaxiaozhi
 * Date 2020/7/10 15:50
 * Desc 演示Kafak-JavaAPI实现Kafak生产者-同步和异步发送消息
 */
public class MyKafkaProducer {
    
    
    public static void main(String[] args) throws Exception {
    
    
        //注意TODO原本表示该任务没有完成,后续需要继续编写,这里使用仅仅为了步骤清晰
        //TODO 1.准备连接参数
        Properties props = new Properties();
        //kafka集群地址
        props.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"node01:9092");
        //消息确认机制--表示是否发送成功
        //0:表示只要发出去就认为发送成功,可能会有数据丢失,一般不用,除非对性能要求特别高,且不在乎数据丢失
        //1:表示只要leaader收到就认为发送成功,开发中可以使用,但是如果对于数据安全要求还是较高,该配置不适合
        //-1/all:表示所有的ISR副本(Leader+Follower)都收到才认为发送成功,也就是必须等到Follower把数据从Leader上同步过来得了
        props.setProperty("acks","all");
        //重试次数
        props.setProperty("retries","2");
        //每次重试间隔多久
        props.setProperty("retries.backoff.ms","20");
        //buffer缓冲区大小,默认32M,单位是byte
        props.setProperty("buffer.memory","10240000");
        //batch批大小,默认16K
        props.setProperty("batch.size","10240");
        //发送batch批的间隔
        props.setProperty("linger.ms","25");
        //每次发送request对象的大小
        props.setProperty("max.request.size","102400");
        //k-v序列化--因为跨网络传输
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

        //指定使用我们自己定义的分区器
        //props.put("partitioner.class", "cn.hanjiaxiaozhi.producer.MyPartitioner");

        //指定使用自定义的拦截器
        props.put("interceptor.classes", Arrays.asList(
                "cn.hanjiaxiaozhi.interceptor.MyInterceptor1_Mosaic",
                "cn.hanjiaxiaozhi.interceptor.MyInterceptor2_Statistics"
        ));

        //TODO 2.根据参数创建生产者对象
        KafkaProducer<String, String> producer = new KafkaProducer<>(props);

        //TODO 3.同步发送
        for (int i = 0; i < 10; i++){
    
    
            ProducerRecord<String, String> record = new ProducerRecord<>("test_topic", "key_"+i, "138888888" + i);
            RecordMetadata metadata = producer.send(record).get();
            System.out.println("同步发送后获得分区编号和offset :"+metadata.partition() + "---" + metadata.offset());
        }


        //TODO 5.关闭资源
        producer.close();
    }
}

Test consumer

package cn.hanjiaxiaozhi.consumer;

import org.apache.kafka.clients.CommonClientConfigs;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

import java.util.Arrays;
import java.util.Properties;

/**
 * Author hanjiaxiaozhi
 * Date 2020/7/11 10:17
 * Desc 演示Kafka-JavaAPI-消费者-自动提交偏移量
 */
public class MyKafkaConsumerAutoCommit {
    
    
    public static void main(String[] args) {
    
    
        //TODO 1.准备连接参数
        Properties props = new Properties();
        ///集群地址
        props.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG,"node01:9092");
        //k-v反序列化
        props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        //是否自动提交偏移量
        props.put("enable.auto.commit","true");//true表示自动提交
        //自动提交的时间间隔
        props.put("auto.commit.interval.ms","1000");
        //指定从哪个位置开始消费
        //1.earliest ：当各分区下有已提交的 Offset 时，从提交的 Offset开始消费；无提交的Offset 时，从头开始消费；
        //2.latest ： 当各分区下有已提交的 Offset 时，从提交的 Offset 开始消费；无提交的 Offset时，消费新产生的该分区下的数据
        //3.none ： Topic 各分区都存在已提交的 Offset 时，从 Offset 后开始消费；只要有一个分区不存在已提交的 Offset，则抛出异常。
        props.put("auto.offset.reset","latest ");
        //设置一个消费者组名称--消费者组是用来管理消费者的,一个组内可以有1~n个消费者
        props.put("group.id","myconsumer");//组名称如果不给的话,会自动生成一个,不方便查看管理

        //TODO 2.根据参数创建消费者
        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

        //TODO 3.订阅主题
        consumer.subscribe(Arrays.asList("test_topic"));


        //TODO 4.开始消费
        //注意:一直消费订阅的主题
        while (true){
    
    
            //从Kafka集群拉取消息
            ConsumerRecords<String, String> records = consumer.poll(100);
            for (ConsumerRecord<String, String> record : records) {
    
    
                System.out.println("分区:"+record.partition()+" 偏移量:"+record.offset()+" key:"+record.key()+" value:"+record.value());
            }
        }
    }
}