Overview
Streaming microservices are mainly used to process a continuous stream of data in real time. Compared with application microservices, it puts forward more technical requirements for developers. The data itself is not like a pure business application. It is relatively abstract and complex, and has higher performance requirements than application microservices in terms of transmission, storage, and analysis.
In terms of design, Spring Cloud Stream is a microservice framework for building message-driven. It is based on Spring Boot to create microservice applications that provide good support for DevOps, and provides a unified, unified, and efficient way to connect and propagate messages through Spring Integration integration tools. Flexible operating platform. The operating platform is configurable, such as configuring producers/consumers, consumer groups to which consumers belong, and even message partitions (requires the support of the message middleware provider itself).
The use of Spring Cloud Stream is very simple. By adding @EnableBindingannotations you can connect the message middleware and create a streaming microservice. @StreamListenerReceived messages can be processed asynchronously by annotating methods .
quick start
The following uses a simple sink application to demonstrate the usage of the message receiver of the Stream application.
@SpringBootApplication
@EnableBinding(Sink.class)
public class VoteRecordingSinkApplication {
public static void main(String[] args) {
SpringApplication.run(VoteRecordingSinkApplication.class, args);
}
@StreamListener(Sink.INPUT)
public void processVote(Vote vote) {
votingService.recordVote(vote);
}
}
@EnableBindingAn annotation can receive one or more input/output channel interface classes as parameters (the parameter of the annotation in this example is only Sinkone interface). Spring Cloud Task provides Source, Sink, and Processorinterfaces by default to define message sources, receivers and processing channels. Definition extension of channel support.
Below is the interface definition code for the Sinkmessage receiver channel.
public interface Sink {
String INPUT = "input";
@Input(Sink.INPUT)
SubscribableChannel input();
}
@InputAnnotations define input channels through which incoming messages can be received. @OutputThe annotation defines the message output channel method, and through this annotation, the channel through which the application publishes the message can be defined. The parameters of these two annotations are the names of the input/output message channels. If no parameters are specified for the annotation, the method name of the method marked by the annotation is used as the channel name by default.
Spring Cloud Stream will automatically create an implementation class for the message channel interface through dynamic proxy technology. The following code is a method to test whether the message channel is successfully created.
@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = VoteRecordingSinkApplication.class)
@WebAppConfiguration
@DirtiesContext
public class StreamApplicationTests {
@Autowired
private Sink sink;
@Test
public void contextLoads() {
assertNotNull(this.sink.input());
}
}
Core idea
Spring Cloud Stream has designed many abstractions and foundational components to simplify message-driven microservices, the core of which is briefly outlined below.
- Spring Cloud Streaming Microservice Model
- Binder abstract message middleware
- Continuous Publish/Subscribe Support
- Support consumer groups
- Support message partitioning
- Pluggable Binder API
1. Streaming Microservices Model
A Spring Cloud Stream application takes the message middleware as the core, and the application communicates channelswith the . channelsCommunicate with external message middleware through a specific Binder implementation. Figure 1 shows the schematic diagram of the Spring Cloud Stream application architecture.
2. Abstraction of message middleware
Spring Cloud Stream provides an abstract Binder for Kafka and Rabbit MQ to represent message middleware, and it also provides middleware for testing TestSupportBinder, which can directly send reliable messages to the channel and perform assertion testing, which can be written using the extended API own Binder.
Spring Cloud Stream uses the general Spring Boot configuration method, and the abstract Binder provides good support for flexible configuration of how to connect message middleware and send messages. For example, message publishers can dynamically select the target (kafka topic or RabbitMQ exchanges) to connect to the channel at runtime. These external property configurations can be given by Spring Boot (through application parameters, environment variables, application.yml or application .properties configuration file).
Spring Cloud Stream will automatically discover and use the binders available on the classpath. You can make the same code support multiple message middleware by relying on different middleware when building the project. In complex application scenarios, you can also specify which Binder to bind to each channel in advance, and package multiple binders directly.
3. Continuous Publish/Subscribe Support
Streaming microservice applications communicate through a publish/subscribe model, and data is propagated through a shared topic. Figure 2 shows the deployment architecture diagram of a set of streaming microservices, which intuitively illustrates the relationship between multiple streaming microservices.
Data passing through the HTTP interface is forwarded to the raw-sensor-datatarget topic. The topic of computing the average time window and persisting data to HDFS is shared by two independent microservices.
This publish-subscribe-based communication model can reduce the complexity of producers and consumers, and can extend the topology logic without breaking existing data flow. For example, you can add an application to monitor and display the highest value after the average time window service, or even add an application to calculate the average error time in the same data stream. Communication between microservices through shared topics greatly reduces the coupling between services than point-to-point communication.
The publish-subscribe model is not a new concept, Spring Cloud Stream just takes some extra steps to make the publish-subscribe model an excellent choice for building microservices. By providing support for the local message intermediate, Spring Cloud Stream can concisely build a cross-platform publish-subscribe model.
4. Consumer Group
The publish-subscribe model makes it easy to communicate and connect between applications through a shared topic, but when deploying multiple application instances for high availability, it is also necessary to prevent the application from repeatedly consuming messages in the topic. There should be a competitive consumption relationship between multiple application instances, and a message should only be consumed by one consumer.
Spring Cloud Stream simulates the above requirements through the concept of consumer groups. Each consumer input channel binder can use spring.cloud.stream.bindings.<channelName>.groupproperties to specify the consumer group to which it belongs.
4.1 Persistent consumer groups
Like Spring Cloud Stream's independently runnable service model, subscriptions to consumer groups are durable. This sentence means that the consumer group to which the binder belongs by default is persistent. As long as a consumer in the consumer group is created, unless all consumers are stopped, the group will Messages will be received continuously.
Usually, when the destination of the message is known, we will specify the consumer group. When upgrading to the Spring Cloud Stream application, you must specify the consumer group to which the US-Russian input binder belongs, so that you can Avoid repeated consumption of messages when deploying multiple application instances.
Partition support
Spring Cloud Stream provides support for message partitioning for applications with multiple producer instances. In the case of partitions, the physically connected media is treated as a multi-partition structure, with one or more producers sending data to multiple consumers, and ensuring that data with certain characteristics is only accessed by a single consumer instance Consumption.
Spring Cloud Stream provides a common abstraction to implement partition processing in a unified manner. The specific partition strategy can be implemented by message middleware that provides a partition mechanism.
Partitioning is an important concept in state processing. To ensure that related data can be processed together by a service, the concept of partitioning is very important, both in terms of performance and consistency.
about
Sample source code
The stream-simple-listener subproject of spring-cloud-stream-learning
postscript
The streaming microservice provided by Spring Cloud Stream is a brand-new concept, which uses message-driven as the service communication method, and performs asynchronous and high-performance real-time processing of data streams. However, it does not use any new technology, but abstracts each message middleware with a beautiful design, shields its internal implementation principles, and makes the communication between services simpler and more friendly.
The content of this article is mainly the translation of the official documents of Spring Cloud Stream Elmhurst, but the author's level is limited, please point out the imperfections. The content used in this project and document is for study and research purposes only, please indicate the source when reprinting or citing. If you have questions or problems with the documentation, please leave me a message in the project or email me at [email protected] My github : https://github.com/weiwei02/ I believe technology can change the world.
Link
- No previous article
- The next article is in preparation