Teach you how to integrate springboot with kafka

The open source China community team made its first live broadcast, telling the story behind the open source China community in the name of sharing."

This article is shared from Huawei Cloud Community " Getting started with springboot+kafka hand in hand ", author: QGS.

Install kafka

Starting the Kafka local environment requires Java 8+ or above

Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action streaming data of consumers in the website.

Kafka startup methods include Zookeeper and Kraft. You can only choose one of the two startup methods and cannot use them at the same time.

Kafka download https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz

Unzip tar -xzf kafka_2.13-3.7.0.tgz

1. Zookeeper starts Kafka (kafka has built-in zookeeper)

Kafka relies on Zookeeper

1. Start Zookeeper 2. Start Kafka

Start using kafka’s own Zookeeper

./zookeeper-server-start.sh ../config/zookeeper.properties &

./zookeeper-server-stop.sh ../config/zookeeper.properties

./kafka-server-start.sh ../config/server.properties &

./kafka-server-stop.sh ../config/server.properties

2. Zookeeper server starts Kafka

Zookeeper server installation

https://zookeeper.apache.org/

https://dlcdn.apache.org/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz

tar zxvf apache-zookeeper-3.9.2-bin.tar.gz

Configure Zookeeper server

cp zoo_sample.cfg zoo.cfg

Start the Zookeeper server

./zkServer.sh start

Modify Zookeeper port

Add content to Zoo.cfg

admin.serverPort=8099

Restart Zookeeper in the apache-zookeeper-3.9.2-bin/bin directory

Zookeeper server starts kafka

/opt/kafka_2.13-3.7.0/bin directory

./kafka-server-start.sh ../config/server.properties &

Kafka configuration file server.properties

3. Use KRaft to start Kafka

UUID universally unique identifier (Universally Unique Identifier)

1, Generated Cluster UUID: ./kafka-storage.sh random-uuid

2. Format the kafka log directory: ./kafka-storage.sh format -t 3pMJGNJcT0uLIBsZhbucjQ -c ../config/kraft/server.properties

3. Start kafka: ./kafka-server-start.sh ../config/kraft/server.properties &

springboot integrates kafka

When creating a topic, if you do not specify the number of partitions for the topic, the default is 1 partition.

Modify the server.properties file

because server.properties

listeners=PLAINTEXT://0.0.0.0:9092

advertised.listeners=PLAINTEXT://192.168.68.133:9092

springboot adds dependency on kafka

<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>

After adding the spring-kafka dependency, springboot automatically assembles the beans of kafkaTemplate.

application.yml configures connection to kafka

spring:
kafka:
bootstrap-servers: 192.168.68.133:9092

producer

Send a message

@Resource
private KafkaTemplate<String,String> kafkaTemplate;

@Test
void kafkaSendTest(){
kafkaTemplate.send("kafkamsg01","hello kafka");
}

consumer

receive messages

@Component 
public class KafkaConsumer { 

@KafkaListener(topics = {"kafkamsg01","test"},groupId = "123") 
public void consume(String message){ 
System. out .println("Received message: "+message) ; 
} 

}

If no groupid is configured

Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: No group.id found in consumer config, container properties, or @KafkaListener annotation; a group.id is required when group management is used.

@Component 
public class KafkaConsumer { 

@KafkaListener(topics = {"kafkamsg01","test"},groupId = "123") 
public void consume(String message){ 
System. out .println("Received message: "+message) ; 
} 

}

Click to follow and learn about Huawei Cloud’s new technologies as soon as possible~