[Kafka of big data] 3. Message sending process of Kafka producer and synchronous and asynchronous sending API

  Send the data transmitted from the outside to the kafka cluster.

1 Sending principle

(1) Create a main() thread, create a producer object, call the send method, and pass the interceptor (optional), serializer, and partitioner.

(2) The partitioner sends data to partitions, each partition creates a queue (partitioning is done in memory), the total memory size is 32M, and the size of each batch is 16K.

(3) The sender thread reads the data in the buffer queue and sends it to the Kafka cluster, and pulls data according to batch.size and linger.ms (that is, pulls data after each batch of data is full or after the set time is up) .

(4) The sender thread pulls data, taking each node as a group. When the first request data is sent to broker1, but the broker does not respond in time, it can still send the second request. There are at most 5 requests that have not been received. Response will not continue to send requests.

(5) The selector connects the input stream and the output stream.

(6) Send data after the link is connected.

(7) After the Kafka cluster receives the data, it performs replica synchronization according to the replica mechanism.

(8) After the Kafka cluster receives the data, it responds according to the response mechanism.

(9) The selector judges based on the message fed back by the Kafka cluster.

(10) If it succeeds, delete the request and clear the data of each corresponding partition in the buffer queue; if it fails, retry and resend the request until it succeeds.

2 List of important parameters of the producer

3 Asynchronous send API

3.1 Ordinary asynchronous sending

(1) Requirement: Create a Kafka producer and send it to Kafka Broker asynchronously
(2) Analysis: Asynchronous sending means sending external data to the buffer queue (whether the data in the buffer queue is sent to the Kafka cluster or not).

Steps:
(1) Create a kafka project and import dependencies in pom.xml:

<dependencies>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>3.0.0</version>
    </dependency>
</dependencies>

(2) Create class: com.astudy.kafka.producer.CustomProducer

package com.study.kafka.producer;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class CustomProducer {
    
    
    public static void main(String[] args) {
    
    
        //0.创建 kafka 生产者的配置对象
        Properties properties = new Properties();

        //给 kafka 配置对象添加配置信息：bootstrap.servers
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"hadoop102:9092,hadoop103:9092");

        // key,value 序列化（必须）：key.serializer，value.serializer
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());


        //1.创建 kafka 生产者对象
        KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);

        //2.调用 send 方法,发送消息
        for (int i = 0; i < 3; i++) {
    
    
            kafkaProducer.send(new ProducerRecord<>("first","test"+i));
        }

        //3.关闭资源
        kafkaProducer.close();
    }
}

(3) Test:
Open the Kafka consumer on hadoop102:

 kafka-console-consumer.sh --bootstrap-server hadoop102:9092,hadoop103:9092 --topic first

Execute the code in IDEA and observe whether the message is received in the hadoop102 console:

3.2 Asynchronous sending with callback function

Analysis:
  The callback function will be called when the producer receives the ack, which is an asynchronous call. This method has two parameters, which are metadata information (RecordMetadata) and exception information (Exception). If Exception is null, it means that the message is sent successfully. If Exception is not null, indicating that the sending of the message failed.

package com.study.kafka.producer;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class CustomProducerCallback {
    
    
    public static void main(String[] args) throws InterruptedException {
    
    
        //0.创建 kafka 生产者的配置对象
        Properties properties = new Properties();

        //给 kafka 配置对象添加配置信息：bootstrap.servers
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"hadoop102:9092,hadoop103:9092");

        // key,value 序列化（必须）：key.serializer，value.serializer
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());


        //1.创建 kafka 生产者对象
        KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);

        //2.调用 send 方法,发送消息
        for (int i = 0; i < 3; i++) {
    
    
            kafkaProducer.send(new ProducerRecord<>("first", "test" + i), new Callback() {
    
    
                // 该方法在 Producer 收到 ack 时调用，为异步调用
                @Override
                public void onCompletion(RecordMetadata recordMetadata, Exception e) {
    
    
                    if (e == null) {
    
    
                        // 没有异常,输出信息到控制台
                        System.out.println("topic:" + recordMetadata.topic() + "  partition:" + recordMetadata.partition());
                    }else {
    
    
                        // 出现异常打印
                        e.printStackTrace();
                    }

                }
            });
            // 延迟一会会看到数据发往不同分区
            Thread.sleep(2);
        }

        //3.关闭资源
        kafkaProducer.close();
    }
}

Test:
Open Kafka consumer on hadoop102:

 kafka-console-consumer.sh --bootstrap-server hadoop102:9092,hadoop103:9092 --topic first

Execute the code in IDEA and observe whether the message is received in the hadoop102 console:

4 Synchronous send API

Analysis: Just call the get() method on the basis of asynchronous sending.

package com.study.kafka.producer;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class CustomProducerSync {
    
    
    public static void main(String[] args) throws ExecutionException, InterruptedException {
    
    
        //0.创建 kafka 生产者的配置对象
        Properties properties = new Properties();

        //给 kafka 配置对象添加配置信息：bootstrap.servers
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,"hadoop102:9092,hadoop103:9092");

        // key,value 序列化（必须）：key.serializer，value.serializer
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());


        //1.创建 kafka 生产者对象
        KafkaProducer<String, String> kafkaProducer = new KafkaProducer<>(properties);

        //2.调用 send 方法,发送消息
        for (int i = 0; i < 3; i++) {
    
    
            kafkaProducer.send(new ProducerRecord<>("first","test"+i)).get();
        }

        //3.关闭资源
        kafkaProducer.close();
    }
}

Test:
Open Kafka consumer on hadoop102:

 kafka-console-consumer.sh --bootstrap-server hadoop102:9092,hadoop103:9092 --topic first

Execute the code in IDEA and observe whether the message is received in the hadoop102 console: