kafka producer

KafkaProducer create a KafkaThread to run Sender.run method.

1. The message sent in the inlet KafkaProducer # doSend but in fact the message is added to the batches:

message is sent by the producer kafka batch, RecordAccumulator class variables have the ConcurrentMap <TopicPartition, the Deque <ProducerBatch >> Batches,
KafkaProducer # doSend current methods would put this message into the ProducerBatch. Then call Sender # wakeup method, try to wake up the blocked io thread.

2. Remove the batches from the data transmission, at the entrance Sender.run, abstract two main logical steps:

NetworkClient.send 2.1
the send here is not a true network to send, first ProducerReuquest Send sequence into an object, and then added to inFlightRequests head, call send selector is actually a KafkaChannel.setSend ()

Send send = request.toSend(nodeId, header);

this.inFlightRequests.add(inFlightRequest);

selector.send(inFlightRequest.send);

2.2 NetworkClient.poll
real network to send

Selector # pollSelectionKeys write processing network event, i.e. sending a message write event and the response is stored in the Selector # completedReceives
producer to send a message, if acks = -1 and 1, i.e., producer request response is required,
in NetworkClient # handleCompletedSends in the not request requires a response, the inFlightRequests removed from
the process in response NetworkClient # handleCompletedReceives
producer ack set value is fixed, in response to need or producer or not a response is required.
To the head of a new request, the received response corresponding to the oldest request, i.e. a request tail.

3. The main classes
KafkaProducer: api directly exposed to the user's class; Sender: key management ProducerBatch
NetworkClient: ProducerBatch object, sent over a network need to be serialized, such connection management, layer closer io
Selector encapsulates the java nio Selector
KafkaChannel

4. ByteBuffer

// ByteBuffer 的使用
// ByteBuffer 初始是写模式
public static void main(String[] args) throws UnsupportedEncodingException {
    // capacity = 512, limit = 512, position = 0
    ByteBuffer buffer = ByteBuffer.allocate(512);
    buffer.put((byte)'h');
    buffer.put((byte)'e');
    buffer.put((byte)'l');
    buffer.put((byte)'l');
    buffer.put((byte)'o');

    // limit = position, position = 0
    buffer.flip (); 

    // get the number of bytes 
    int len = buffer.remaining ();
     byte [] = DST new new  byte [len]; 
    buffer.get (DST); 
    System.out.println ( new new String (DST) );
     // Conclusion: ByteBuffer just byte [] package 
} 

// the SocketChannel
 // output
 // the SocketChannel # Write (java.nio.ByteBuffer,)
 // read the input
 // the SocketChannel # read (java.nio.ByteBuffer,)

Guess you like