[01] Kafka producer

0. Contents

Message in step 1. Production

• Producers client configuration parameters, create instances producer
• Construction of message to be sent
• Send a message
• Close producers

2. ProducerRecord property

public class ProducerRecord<K, V> {
    private final String topic;//必填项
    private final Integer partition;
    private final Headers headers;
    private final K key;
    private final V value;//必填项
    private final Long timestamp;
}

key used to calculate the partition number, sent to the specified partition is determined

3. The necessary parameters

• bootstrap.servers: Specifies the address of the broker kafka cluster is empty by default, although you can find other broker from a given broker, but in order to prevent a node goes down resulting in a failure message is sent, it is recommended to fill in at least two broker address.
• key.serializer、value.serializer
• client.id: Client client ID, our if set, will automatically generate a non-empty string.
• Setting Tips:
  attribute name bad record, you can use ProducerConfig constants class.

4. Send message

• Hair after forget (fire-and-foget)
  the highest performance, reliability worst

try {
    producer.send(record);
} catch (Exception e) {
    e.printStackTrace();
}

• Synchronization (sync)
  difference in performance of the synchronous transmission is required block waiting after the transmission is completed, go to send the next message.
  The method itself is asynchronous send, Future object returned by the method using the get synchronized.

try {
    producer.send(record).get();
} catch (Exception e | InterruptedException e) {
    e.printStackTrace();
}

The Future.get if desired () return value, the following manner may be employed. Returns the value contained in the message subject matter, the partition number, an offset, the time stamp and other metadata information.

try {
    Future<RecordMetadata> future = producer.send(record);
    RecordMetadata metadata = future.get();
    System.out.println(metadata.topic() + "-" + 
        metadata.partition() + "-" + metadata.offset());
} catch (Exception e | InterruptedException e) {
    e.printStackTrace();
}

Since the send () method returns a Future object, you can use java concurrency related methods to achieve rich.
Synchronous transmission can configure the number of client retry parameters, if the number of retries is exceeded, the exception must be handled.

• Asynchronous (the async)
  Future send(ProducerRecord<K, V> record, Callback callback);

producer.send(record, new Callback() {
    @Override
    public void onCompletion(RecordMetadata metadata, Exception exception) {
        if (exception != null) {
            exception.printStackTrace();
        } else {
            System.out.println(metadata.topic() + "-" + 
                metadata.partition() + "-" + metadata.offset());
        }
    }
})

kafka message partition ensuring orderly circumstances, the callback function can ensure the orderly.

producer.close();
public void close(long timeout, TimeUnit timeUnit); //等待超时时间之后，强行退出。

close () method blocks waiting for the completion of all of the messages are sent and then closed.

5. The partitioner

• Message specified weight partition field, according to a corresponding partition of the field
• Partition field is not specified, according to the message body key field, assigned by the partitioner partitions
  kafa use partitioner i.e. client class DefaultPartitioner

DefaultPartitioner implements Partitioner {
    public int partition(...);  //计算分区号
    public void close();    //关闭分区器时释放相应资源，空方法
}

• Default partition is:
  (1) if the key is not null, the hash key obtained for the partition number ( any one of all partitions, whether or not available ).
  (2) key is null, the polling is sent to each available partition , if no partition is available, the poll to each partition.
• How to customize the partitioner?
  (1) interfaces implemented Partitioner, overwriting Partition () Method
  (2) Configuration parameters client

props.put(ProducerConfig.PARTITIONER_CLASS_CONFIG, DemoPartitinoer.class.getName());

6. interceptor

• Role:
  preparatory work (eg: filter messages, modify part of the contents) before sending
  Before sending the callback logic, do some customization needs (eg: statistics)

• Custom interceptor
  (1) implemented ProducerInterceptor <String, String> interfaces, rewriting OnSend () method, rewriting onAcknowledgement () Method
  (2) Configuration parameters client: ProducerConfig.INTECERPTOR_CLASSES_CONFIG
  can specify a plurality of, in the middle of the class name "" when connected, a plurality of interceptors, one fails, then the next one from the interceptor a successful start

  7. overall architecture

  Main thread: KafkaProducer -> interceptor (not necessarily, generally do not) -> device serial number (required) -> partitioner (The key optional) -> Accumulator message -> Sender thread
  Sender threads: SENDER -> Create the request -> submitted to transmit Selector -> not received in response to the request according to the node corresponding to the node are placed in InFlightRequests.
  Sender thread, the first partition information into the node information, so the network layer to the application logic conversion layer IO, and then converts the message protocol to satisfy kafka Request.
  

By how much the size of the configuration parameters max.in.flight.requests.per.connection InFlightRequests comparison with the messages in InFlightRequests, to determine the load condition of each node.

Metadata (information about the partition number of themes, address a copy of the node where the leader, ports, etc) are updated by the Sender thread, select the least loaded node transmits MetaDataRequest get. The default is updated every five minutes.
The main thread also requires the use of Sender updated metadata.

8. Several important parameters producers

• acks -> Under what circumstances considered successfully transmitted
  Default: 1, as long as the leader that is successfully written to success. It is a compromise between reliability and throughput.
  0, that is successful is sent, regardless of whether the server returns the corresponding. The maximum throughput.
  -1 or all, copies of all the ISR successfully. The highest reliability, but it does not guarantee reliable, because the ISR could only leader, in this case the effect is the same 1.
• max.request.size
  maximum message that can be sent to the producer, the default 1MB

9. producers are thread-safe

end ：）