Using Spring Cloud Stream to integrate message middleware: Simplify message processing and asynchronous processing

I. Overview

1 What is Spring Cloud Stream

Spring Cloud Stream is a framework for building message-driven microservices. It encapsulates the interaction with the message middleware and provides a consistent programming model; it avoids the need for developers to pay attention to the details of the underlying message middleware.

2 Relationship between Spring Cloud Stream and message middleware

Spring Cloud Stream can be integrated with a variety of different messaging middleware, including RabbitMQ, Kafka, AWS Kinesis, etc. By adding Spring Cloud Stream-related dependencies to the application, we can easily switch between different message middleware at the code level without modifying other codes.

2. The core concept of Spring Cloud Stream

1 Binder

Binder is one of the core components of Spring Cloud Stream and is a bridge connecting message middleware and applications. By configuring Binder, we can specify which message middleware the application uses. Binder's other functions include serialization and deserialization of messages, flow control (backpressure), error handling, etc.

In the application we can use @EnableBindingannotations to specify the binder.

@SpringBootApplication
@EnableBinding(SampleBinding.class)
public class SampleApplication {
    
    
    public static void main(String[] args) {
    
    
        SpringApplication.run(SampleApplication.class, args);
    }
}

interface SampleBinding {
    
    
    @Input
    MessageChannel input();

    @Output
    MessageChannel output();
}

2 Destination

Destination can be understood as the target location for sending or receiving messages. In Spring Cloud Stream, Destination is parsed by DestinationResolver. It usually includes destination name (name), group name (group name) and other information.

3 Channel

A Channel is an endpoint used in an application to send or receive messages. There are three main types of Channel in Spring Cloud Stream, namely Source, Sink and Processor.

Source Channel (generally used for message producers)

Use @Outputannotations , and when sending a message to this Channel, the message will be automatically sent to the corresponding target address of the message middleware.

For example:

@EnableBinding(Source.class)
public class SampleSource {
    
    

    @Autowired
    private Source source;

    @Scheduled(fixedDelay = 1000L)
    public void sendMessage() {
    
    
        this.source.output().send(MessageBuilder.withPayload(new Message("hello")).build());
    }

}

Sink Channel (generally used for message consumers)

Use @Inputannotations . When a new message arrives in the application, it will automatically receive the message from the MessageChannel and make it available for the application to process.

For example:

@EnableBinding(Sink.class)
public class SampleSink {
    
    

    @ServiceActivator(inputChannel = Sink.INPUT)
    public void receiveMessage(Message<String> message) {
    
    
        // handle message payload here
    }

}

Processor Channel (both producer and consumer of messages)

Processor Channel can be seen as a superset of Source Channel and Sink Channel, which can write data (production) and read data (consumption).

For example:

@EnableBinding(Processor.class)
public class SampleProcessor {
    
    

    @Transformer(inputChannel = "input", outputChannel = "output")
    public String transform(String payload) {
    
    
        return payload.toUpperCase();
    }

}

4 Source and Sink

Spring Cloud Stream provides encapsulated Source and Sink types to simplify development. When used in an application, use @EnableBindingannotations to bind Source or Sink to the corresponding Binder. For example, Sink is used to consume messages, the sample code is as follows:

@EnableBinding(Sink.class)
public class SampleSink {
    
    

    @StreamListener(Sink.INPUT)
    public void receive(Message<String> message) {
    
    
        // handle message here
    }

}

And Source is used to produce messages, the sample code is as follows:

@EnableBinding(Source.class)
public interface SampleSource {
    
    

    @Output
    MessageChannel output();

}

@Service
public class MyService {
    
    

    private final SampleSource source;

    public MyService(SampleSource source) {
    
    
        this.source = source;
    }

    public void someMethod() {
    
    
        this.source.output().send(MessageBuilder.withPayload("hello world").build());
    }
}

3. Basic usage process of Spring Cloud Stream

1 Preparations

Introduce spring-cloud-starter-stream-{binder} dependency in a Spring Boot application (where {binder} represents the message middleware used, such as Kafka, RabbitMQ, etc.) If you need to send and receive messages, you also need to introduce spring -cloud-stream.

2 Define the message channel

Define input and output channels by defining @Input and @Output annotations, for example:

public interface MyProcessor {
    
    
    String INPUT = "my-input";
    String OUTPUT = "my-output";
    
    @Input(INPUT)
    SubscribableChannel input();
    
    @Output(OUTPUT)
    MessageChannel output();
}

The above code defines a MyProcessor interface with an input channel named "my-input" and an output channel named "my-output".

3 Send and receive messages using Source and Sink

By using the Source and Sink interfaces provided by Spring Cloud Stream, we can easily send and receive messages. For example:

@Autowired
private Source source;

@Autowired
private Sink sink;

...

source.output().send(MessageBuilder.withPayload("hello").build());

String message = (String) sink.input().receive().getPayload();

In the above code example, a Source and Sink instance is automatically assembled through the @Autowired annotation, and the channels for sending and receiving messages are specified in the output() method and input() method, respectively.

4 Configure the integration of Binder and message middleware

In the application.properties or application.yml configuration file, you can configure Binder related properties through the spring.cloud.stream.{binder}.xxx configuration item. At the same time, it is also necessary to specify the relevant information of the message middleware, as shown in the following example:

# Kafka配置示例
spring.cloud.stream.kafka.binder.brokers=localhost:9092
spring.cloud.stream.kafka.binder.auto-create-topics=true
spring.cloud.stream.kafka.binder.configuration.foo=bar

# RabbitMQ配置示例
spring.cloud.stream.rabbit.bindings.my-output.destination=my-exchange
spring.cloud.stream.rabbit.bindings.my-output.producer.routing-key-expression=headers['myKey']
spring.cloud.stream.rabbit.bindings.my-input.destination=my-queue
spring.cloud.stream.rabbit.bindings.my-input.consumer.bindingRoutingKey=my-routing-key

4. Optimization of message processing and asynchronous processing flow

1 Message Segmentation and Batch Processing

Using Spring Cloud Stream, you can realize message segmentation and batch processing by configuring related parameters. For details, please refer to the relevant documents of Binder.

2 Message processing method based on functional programming model

In the processing method based on the functional programming model, you can define a Function interface and write message processing logic in it, for example:

@Bean
public Function<String, String> uppercase() {
    
    
    return String::toUpperCase;
}

In the above code example, we defined a Bean named uppercase, whose type is Function<String, String>, which converts the input string to uppercase and outputs it.

3 Message processing method based on reactive programming model

In the processing method based on the reactive programming model, you can use the related classes and interfaces provided by reactive-streams or reactor to process messages asynchronously. For details, please refer to the relevant documents of Spring Cloud Stream.

Five, Spring Cloud Stream common problems and solutions

1 Selection and configuration of Binder

Binder is the core component of Spring Cloud Stream, which realizes the interaction with MQ middleware. Spring Cloud Stream supports multiple binders, such as RabbitMQ, Kafka, etc. We need to select a suitable Binder according to the actual situation and configure it accordingly.

1.1 Binder selection

The following factors need to be considered when choosing a Binder:

• Application dependency on MQ middleware
• Performance and reliability of MQ middleware
• development and maintenance costs

1.2 Binder configuration

Binder configuration includes general configuration and specific Binder configuration. The general configuration is as follows:

spring.cloud.stream:
  bindings:
    input: #input定义
      destination: inputTopic #指定发送到哪个Topic
      content-type: application/json #消息类型
    output: #output定义
      destination: outputTopic #指定发送到哪个Topic
      content-type: application/json #消息类型
  binders:
    binder1: #binder定义
      type: rabbit
      environment:
        spring:
          rabbitmq:
            host: rabbit-server-host #RabbitMQ服务器主机名或IP地址
            port: 5672 #RabbitMQ服务器端口
            username: guest #用户名
            password: guest #密码

2 Problems of message loss and repeated consumption

In actual business, problems of message loss and repeated consumption may be encountered. In order to solve these problems, the following methods can be adopted:

• Persistent messages: For important messages, they can be persisted to the disk. In the event of failures such as downtime, the messages will not be lost.
• Message deduplication: You can avoid repeated consumption by recording the message IDs that consumers have already consumed on the consumer side.
• Manual ACK: Change the consumption mode from automatic ACK to manual ACK to ensure that ACK confirmation is performed after the message is correctly processed to avoid repeated consumption.

3 How to monitor and tune message processing performance

In order to ensure the high performance and reliability of the application, it is necessary to monitor and tune the message processing performance. The following methods can be used:

• Monitoring indicators: You can use the monitoring indicators provided by Spring Cloud Stream to monitor the sending and consumption of messages, delay and other information.
• Tuning parameters: You can adjust parameters such as message batch size and thread pool size according to business needs to improve processing performance.

6. Case Analysis

1 Use Spring Cloud Stream to integrate RabbitMQ to implement message queue

//引入相关依赖
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rabbit</artifactId>
</dependency>

//定义消息发送接口
public interface MessageSender {
    
    
    @Output("myMessageChannel")   //使用@Output注解声明消息通道名称
    MessageChannel output();
}

//定义消息接收接口
public interface MessageReceiver {
    
    
    @Input("myMessageChannel")    //使用@Input注解声明消息通道名称
    SubscribableChannel input();
}

//使用@EnableBinding注解启用绑定功能，连接RabbitMQ
@SpringBootApplication
@EnableBinding({
    
    MessageSender.class, MessageReceiver.class})
public class RabbitMQApplication {
    
    
    public static void main(String[] args) {
    
    
        SpringApplication.run(RabbitMQApplication.class, args);
    }

    //在controller中注入消息发送接口，并调用output()方法发送消息
    @Autowired
    private MessageSender messageSender;

    @GetMapping("/send")
    public void sendMessage() {
    
    
        String message = "Hello RabbitMQ!";
        messageSender.output().send(MessageBuilder.withPayload(message).build());
    }

    //在Service中注入消息接收接口，并使用@StreamListener注解监听消息
    @Autowired
    private MessageReceiver messageReceiver;

    @StreamListener("myMessageChannel")
    public void receiveMessage(String message) {
    
    
        System.out.println("Received message: " + message);
    }
}

2 Use Spring Cloud Stream to integrate Kafka to realize message stream processing

//引入相关依赖
<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

//定义消息发送接口
public interface MessageSender {
    
    
    @Output("myMessageChannel")   //使用@Output注解声明消息通道名称
    MessageChannel output();
}

//定义消息接收接口
public interface MessageReceiver {
    
    
    @Input("myMessageChannel")    //使用@Input注解声明消息通道名称
    SubscribableChannel input();
}

//使用@EnableBinding注解启用绑定功能，连接Kafka
@SpringBootApplication
@EnableBinding({
    
    MessageSender.class, MessageReceiver.class})
public class KafkaApplication {
    
    
    public static void main(String[] args) {
    
    
        SpringApplication.run(KafkaApplication.class, args);
    }

    //在controller中注入消息发送接口，并调用output()方法发送消息
    @Autowired
    private MessageSender messageSender;

    @GetMapping("/send")
    public void sendMessage() {
    
    
        String message = "Hello Kafka!";
        messageSender.output().send(MessageBuilder.withPayload(message).build());
    }

    //在Service中注入消息接收接口，并使用@StreamListener注解监听消息
    @Autowired
    private MessageReceiver messageReceiver;

    @StreamListener("myMessageChannel")
    public void receiveMessage(String message) {
    
    
        System.out.println("Received message: " + message);
    }
}

7. Summary and review

Advantages and disadvantages of Spring Cloud Stream

advantage:

• It simplifies the complexity of using the message middleware and improves the development efficiency.
• Supports a variety of message middleware with strong flexibility.
• Provides a consistent programming model, making applications easier to expand and upgrade.

shortcoming:

• For some advanced configurations and functions, a certain understanding of messaging middleware is still required.

• Runtime performance may be affected somewhat.