An application needs to be written in many cases to Kafka messages: record the user's activity (for audit and analysis), metrics record, save log messages, recording information intelligent appliances, asynchronous communication with other applications, the buffer is about data written to the database, and so on.
A variety of usage scenarios imply diverse needs: whether each message is important? Whether to allow the loss of a small part of the message? Whether occasional duplicate messages can be accepted? Is there a strict latency and throughput requirements?
A direct impact on different usage scenarios will have on the use and configuration Producers API.
Message sending process
First create a ProducerRecord object.
ProducerRecord target object contains the theme and content to be sent.
You can specify key or partition.
If there is a key word, the key sequence into a byte array, for transmission over the network. Similarly, content to be transmitted, i.e. values need serialized byte array.
Next, the data is passed to the partitioner.
If before ProducerRecord objects in the specified partition, then the partition is does not do anything directly to the specified partition to return.
If you do not specify a partition, then the partition will be selected based on a partition ProducerRecord object key.
After selecting the partition, the producers know which topics and partitions to send the records.
Then, this record is added to a record in the batch.
All messages inside this batch will be sent to the same theme and partitions.
There is a separate thread is responsible for these batch records sent to the appropriate broker.
Upon receipt of these messages in the server returns a corresponding.
If the message is successfully written to Kafka, it returns a RecordMetaData object that contains the theme and partition information, and records the offset of the partition.
If the write fails, it will return an error.
Producers after receiving the error will attempt to resend the message, after several failed or if it returns an error message.
Kafka producers parameters
bootstrap.servers
This attribute specifies the broker address list, the address format is host: port.
Listing broker need not contain all the addresses from producer looks for in a given broker to other broker of information.
It recommended that at least two broker of information, once one of the stand-alone cluster, the producers are still able to connect to.
This parameter is a mandatory parameter.
key.serializer
the key broker and desired values are byte array of the received message.
Producers interface allows the use of parameterized types, can be transmitted to the broker as a Java object keys and values. This code has good readability, but producers need to know how to put these into a Java object into a byte array.
key. must be set to a serializer org.apache.kafka.common.serialization.Serializer class that implements the interface, the producer will use the key sequence to the class into a byte array.
Kafka client provides ByteArraySerializer, StringSerializer and IntegerSerializer default.
This parameter must be set, even if only the value sent.
value.serializer
The sequence of values of a specified class.
This parameter must be set.
Send a message
Producers can use a single thread single consumer, consumers can use a single multiple threads.
Or increase consumer.
Send and forget
We send messages to the server, but does not care if it's normal reach.
In most cases, the message will reach normal, because Kafka is highly available, and the producers will try to resend automatically.
However, this approach sometimes losing messages.
Synchronous transmission
We use the send () to send a message, it will return a Future object, call the get () method to wait. We can know whether the message is sent successfully.
Asynchronous Transmission
We call the send () method and specify a callback function, the function is called when the server returns a response.