Article Directory
I. Introduction
1.1 Conclusion
I like to put the conclusion first and explain it later.
- Synchronous writing, waiting for the result to return:
SendResult<String, String> stringStringSendResult = kafkaTemplate.send("cancan", "5", msg).get();
- Asynchronous writing, do not wait for the result to return:
ProducerRecord<String, String> record = new ProducerRecord<>("xixi", "5", msg);
ListenableFuture<SendResult<String, String>> future = kafkaTemplate.send(record);
future.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onFailure(@NonNull Throwable throwable) {
//错误结果
}
@Override
public void onSuccess(SendResult<String, String> result) {
//正确结果
}
});
2.1 Comparison
When some students use rocketmq, synchronous sending and asynchronous sending are clear at a glance, as shown in the figure below
When it comes to springboot kafka, some students are confused. As shown in the figure below, it is not obvious which one is synchronous and which one is asynchronous.
2. Case
2.1 Dependence
I am using springboot 2.2.1
. The dependencies of kafka are:
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
2.2 Directly present the case
Send synchronously
@Autowired
private KafkaTemplate<String, String> kafkaTemplate;
//get的时候，为同步
@RequestMapping("/sendsyn")
public String sendSyn() throws ExecutionException, InterruptedException {
//一个400k的字符串
String msg = FileUtil.fileToBase64("test2");
long s1 = System.currentTimeMillis();
SendResult<String, String> stringStringSendResult = kafkaTemplate.send("cancan", "5", msg).get();
ProducerRecord<String, String> producerRecord = stringStringSendResult.getProducerRecord();
System.out.println(producerRecord.toString());
long s2 = System.currentTimeMillis();
System.out.println("同步耗时："+ (s2 - s1)); //同步耗时长：312
return "同步发送";
}
Asynchronous send 1
//不为get,为异步
@RequestMapping("/sendasy")
public String sendAsy() throws ExecutionException, InterruptedException {
//一个400k的字符串
String msg = FileUtil.fileToBase64("test2");
long s1 = System.currentTimeMillis();
ListenableFuture<SendResult<String, String>> cancan = kafkaTemplate.send("cancan", "5", msg);
cancan.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onFailure(@NonNull Throwable throwable) {
System.out.println("结果失败");
}
@Override
public void onSuccess(SendResult<String, String> result) {
ProducerRecord<String, String> producerRecord = result.getProducerRecord();
System.out.println(producerRecord.toString());
long s4 = System.currentTimeMillis();
System.out.println("结果成功,耗时："+(s4 - s1)); //结果成功,耗时：521
}
});
long s2 = System.currentTimeMillis();
System.out.println("异步耗时："+ (s2 - s1)); //异步耗时长：3
return "异步发送";
}
Asynchronous send 2
//异步的其他写法
@RequestMapping("sendasy2")
public String sendasy2() throws ExecutionException, InterruptedException {
//一个400k的字符串
String msg = FileUtil.fileToBase64("test2");
ProducerRecord<String, String> record = new ProducerRecord<>("cancan", "5", msg);
long s1 = System.currentTimeMillis();
ListenableFuture<SendResult<String, String>> cancan = kafkaTemplate.send(record);
cancan.addCallback(new ListenableFutureCallback<SendResult<String, String>>() {
@Override
public void onFailure(@NonNull Throwable throwable) {
System.out.println("结果失败");
}
@Override
public void onSuccess(SendResult<String, String> result) {
ProducerRecord<String, String> producerRecord = result.getProducerRecord();
System.out.println(producerRecord.toString());
long s4 = System.currentTimeMillis();
System.out.println("结果成功,耗时："+(s4 - s1)); //结果成功,耗时：521
}
});
long s2 = System.currentTimeMillis();
System.out.println("异步耗时："+ (s2 - s1)); //异步耗时长：2
return "异步发送2";
}
2.3 Summary, Kafka's send is sent asynchronously, calling get() to achieve synchronization
It can be seen from the above that Kafka's send is sent asynchronously. If you want to be synchronous, you can call the get method and wait for the result to appear.
3. Off-topic
3.1 message.max.bytes
By default, the kafka server only receives a piece of data with a size of 1m.
If you want to send data larger than 1m, you need to set the
server configuration
message.max.bytes = 20000000. The
default is 1000000.
The maximum size of the message that the server can receive. The important thing is that the settings of this property between the consumer and the producer must be synchronized, otherwise the message published by the producer is too big for the consumer
3.2 max.request.size
The client configuration
max.request.size = 20000000
defaults
to the maximum number of bytes requested by 1028576 . This is also an effective coverage of the maximum record size. Note: The server has its own coverage of the message record size, which is different from this setting. This setting will limit the number of batch requests sent by the producer each time to prevent a huge number of requests.
3.3 File transfer to base64 class
package com.cat.demo.kafkaspringboot.utils;
import org.apache.tomcat.util.codec.binary.Base64;
import java.io.*;
//需要的jar包名commons-codec-1.6.jar
/**
* 文件和Base64之间的相互转化工具类
* @author rmk
*/
public class FileUtil {
/**
*
* @param path 文件全路径(加文件名)
* @return String
* @description 将文件转base64字符串
* @date 2019年11月24日
* @author rmk
*/
public static String fileToBase64(String path) {
String base64 = null;
InputStream in = null;
try {
File file = new File(path);
in = new FileInputStream(file);
byte[] bytes = new byte[(int) file.length()];
in.read(bytes);
base64 = new String(Base64.encodeBase64(bytes),"UTF-8");
System.out.println("将文件["+path+"]转base64字符串:"+base64);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (in != null) {
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return base64;
}
/**
* @param outFilePath 输出文件路径, base64 base64文件编码字符串, outFileName 输出文件名
* @return String
* @description BASE64解码成File文件
* @date 2019年11月24日
* @author rmk
*/
public static void base64ToFile(String outFilePath,String base64, String outFileName) {
System.out.println("BASE64:["+base64+"]解码成File文件["+outFilePath+"\\"+outFileName+"]");
File file = null;
//创建文件目录
String filePath=outFilePath;
File dir=new File(filePath);
if (!dir.exists() && !dir.isDirectory()) {
dir.mkdirs();
}
BufferedOutputStream bos = null;
java.io.FileOutputStream fos = null;
try {
byte[] bytes = Base64.decodeBase64(base64);
file=new File(filePath+"/"+outFileName);
fos = new java.io.FileOutputStream(file);
bos = new BufferedOutputStream(fos);
bos.write(bytes);
} catch (Exception e) {
e.printStackTrace();
} finally {
if (bos != null) {
try {
bos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (fos != null) {
try {
fos.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
}