1. Principle of interceptor
- Producer interceptor (interceptor) was introduced in Kafka 0.10 version, mainly used to implement customized control logic on the client side.
- For the producer, the interceptor gives users the opportunity to make some customized requirements for the message before the message is sent and before the producer's callback logic, such as modifying the message. At the same time, the producer allows users to specify multiple interceptors to act on the same message in sequence to form an interceptor chain.
- Intercetpor's implementation interface is org.apache.kafka.clients.producer.ProducerInterceptor, and its defined methods include:
(1) configure(configs) : Called when obtaining configuration information and initializing data.
(2) onSend(ProducerRecord) : This method is encapsulated in the KafkaProducer.send method, that is, it runs in the user's main thread. The Producer ensures that this method is called before the message is serialized and the partition is calculated. The user can do any operation on the message in this method, but it is best to ensure that the topic and partition to which the message belongs are not modified, otherwise it will affect the calculation of the target partition.
(3) onAcknowledgement(RecordMetadata, Exception) : This method will be called after the message is successfully sent from the RecordAccumulator to the Kafka Broker or when the sending process fails. And usually before the producer callback logic is triggered. onAcknowledgement runs in the producer's IO thread, so do not put heavy logic in this method, otherwise it will slow down the producer's message sending efficiency.
(4) close : close the interceptor, mainly used to perform some resource cleanup work. As mentioned earlier, interceptor may be run in multiple threads, so users need to ensure thread safety by themselves during specific implementation. In addition, if multiple interceptors are specified, the producer will call them in the specified order, and only capture the exceptions that may be thrown by each interceptor and record them in the error log instead of passing them upward. This requires special attention during use.
Second, the interceptor case
1. Preparation
- Create a maven project on the IDE, add dependencies to the pom file
<!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>1.1.1</version>
</dependency>
- Start zookeeper cluster
bin/zkServer.sh start
- Start the kafka cluster
bin/kafka-server-start.sh -daemon config/server.properties
- Kafka cluster opens a consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic bigdata
--from-beginning
2. Demand analysis
Implement a simple interceptor chain composed of dual interceptors. The first interceptor will add the timestamp information to the top of the message value before the message is sent; the second interceptor will update the number of successfully sent and failed messages after the message is sent.
3. Code display
3.1、 增加时间戳拦截器
import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Map;
public class MyTimeInterceptor implements ProducerInterceptor<String,String> {
@Override
public void configure(Map<String, ?> configs) {
}
@Override
public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
/* 取出value值 */
String value = record.value();
/* 创建一个新的 record，把时间戳写入消息体的最前部 */
return new ProducerRecord(record.topic(),record.partition(),record.key(),System.currentTimeMillis() + "---" + value);
}
@Override
public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
}
@Override
public void close() {
}
}
3.2、增加统计次数拦截器，统计发送消息成功和发送失败消息数，并在 producer 关闭时打印这两个计数器
import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Map;
public class MyCountInterceptor implements ProducerInterceptor<String,String> {
private int success;
private int error;
@Override
public void configure(Map<String, ?> configs) {
}
@Override
public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
return record;
}
@Override
public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
/* 统计成功和失败的次数 */
if (metadata != null){
success ++;
}else {
error ++ ;
}
}
@Override
public void close() {
/* 保存结果 */
System.out.println("success: " + success);
System.out.println("error: " + error);
}
}
3.3、producer 主程序
import java.util.ArrayList;
import java.util.Properties;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
public class MyProducerInterceptor {
public static void main(String[] args) {
/* 相当于map操作 */
Properties properties = new Properties();
/* kafka 集群，broker-list */
properties.put("bootstrap.servers", "centos7-1:9092");
/* 等待所有副本节点的应答 */
properties.put("acks", "all");
/* 重试次数 */
properties.put("retries", Integer.valueOf(3));
/* 批次大小 */
properties.put("batch.size", Integer.valueOf(16384));
/* 等待时间 */
properties.put("linger.ms", Integer.valueOf(1));
/* RecordAccumulator 缓冲区大小 */
properties.put("buffer.memory", Integer.valueOf(33554432));
/* key序列化 */
properties.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
/* value序列化 */
properties.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
/* 构建拦截器链 */
ArrayList<String> interceptors = new ArrayList<>();
interceptors.add("com.jh.interceptor.MyTimeInterceptor ");
interceptors.add("com.jh.interceptor.MyCountInterceptor ");
/* 添加拦截器链 */
properties.put("interceptor.classes", interceptors);
/* 创建生产者对象 */
KafkaProducer<String, String> kafkaProducer = new KafkaProducer(properties);
/* 发送消息 */
for (int i = 0; i < 10000; i++){
/* 发送的主题、key、value */
kafkaProducer.send(new ProducerRecord("bigdata", "jh","jh==" + i));
}
/* 一定要关闭 producer，这样才会调用 interceptor的 close 方法 */
kafkaProducer.close();
}
}
4. Operation effect display
