Event-driven architecture in hybrid cloud

Introducing event-based architecture

Coupon network m.cps3.cn

译自：Introduction to Event-Driven Architecture

Several articles related to EDA will be introduced later to fully grasp the advantages and disadvantages of the EDA architecture.

table of Contents

Introducing event-based architecture
- Simple definition
  - Events that never happen
  - Channel transmission event
  - Decoupling through asynchrony and versatility
- The way the event is handled
  - Discrete event processing
  - Event stream processing
  - Complex event handling
- When to use EDA
- Benefits of EDA
- Disadvantages of EDA
- Points to note
- to sum up

In the previous introduction to microservices, the granularity of services and the need to ensure loose coupling were discussed. The article proposes that services should be autonomous, complete and independent, and minimize synchronous communication. Today, we will discuss what loose coupling means and explore an event-driven architecture that is becoming more and more popular in the microservices community.

Simple definition

Event Driven Architecture (EDA) is a software architecture specification that promotes the production and consumption of events.

An event represents an action of interest. Generally, an event corresponds to an action that creates or modifies the state of certain entities. For example, placing an order in an e-commerce application is an event, and distributing an ordered product is also an event. It is also an event that a consumer submits a review of a received product.

Events that never happen

The strange thing about events is that they are not clearly communicated to specific services that might care about them. The event "only happens". What's more important is that events will only happen purely, regardless of whether there are specific services that care about these events. This sounds like a philosophical idea that is often quoted: "If a tree in a forest, no one hears it, will it make a sound?". But this is also the reason why the event is so powerful-the event is converted into a (self-contained) record of something that is happening, and the event and its extensions are (fundamentally speaking) separate from their handlers. In fact, the producer of the event record does not know who the consumer is, or even whether there is a consumer.

A record usually contains information describing an event. In the previous order as an example, the JSON description of the corresponding event is as follows:

{
  "orderId": "760b5301-295f-4fec-95f8-6b303a3b824a",
  "customerId": 28623823,
  "productId": 31334,
  "quantity": 1,
  "timestamp": "2021-02-09T11:12:17+0000"
}

Node: Although there are subtle differences between records and events, they are often interchangeable, that is, the term "event" usually refers to the "record" of an event. To simplify the description, these two terms will be used freely in this article.

The above is a high degree of simplification of an order. The application that initiated the order (shopping cart service) does not know who (how and why) processed the order. The producer will ensure that potential consumers can capture all the information needed to process the event. In other words, the order record may not strictly contain every attribute required to fulfill the order. For example, it is not necessary to directly specify information such as product size, storage location, and consumer's delivery address, but it can be parsed to obtain this information indirectly through the ID in the captured order record. The concept of foreign keys in relational databases also applies to events.

Channel transmission event

If the producer and the consumer do not perceive each other, how should the two communicate?

The answer is to glue through the term "record". Events are usually persisted to a well-known location called the log (sometimes the term "ledger" is also used). The log is at the bottom level, and the event data structure saved by the producer can only be attached to a place that can be accessed by subsequent consumers. Brokers (persistent middleware between producers and consumers) are responsible for operating logs. Once an event is generated, anyone can consume the event.

When dealing with event-driven systems, we often use the term "stream" to describe one or more log interfaces. Log is a physical concept (implemented using files), and a stream is a logical concept, representing a set of unbounded records that constitute an event, but the records must follow a certain order. Different streaming platforms may use proprietary names to refer to a stream. Apache Kafka uses topics and partitions to describe flows.

The relationship between producer, consumer and stream is as follows:

Event-Driven Architecture Reference Model

Recall related concepts:

Events are actions of interest that occur at discrete points in time: they may be observed and described from the outside.
Event persistence as a record: Although the event and the record are related, they are technically different. An event represents the occurrence of something (such as a state change) and is intangible in itself. And a record is an accurate description of the event. We usually use the term "event" to refer to its corresponding record.
The producer is the receiver that detects the event by publishing the corresponding record to the stream. ( Posting a record means that an event has occurred )
A stream is an ordered record of persistence. They are usually persisted by one or more disk-based logs. Of course, database tables, distributed consensus protocols, or even blockchain-style distributed ledgers can also be used to support persistence.
Brokers are responsible for accessing the stream, facilitating read and write operations, processing consumer status, and performing various "housekeeping" on the stream. For example, a broker may intercept the content of the stream when the record overflows.
The consumer reads the stream and then responds to the received record. The consumer's response to the event may be accompanied by some additional operations. For example, a consumer may persist an entry in the local database (reconstruct the state of the remote entity by issuing an "update" event) ( that is, update the description of the remote entity ).
Consumers and producers may overlap. For example, the responder to the event may also generate one or more derived events.

Decoupling through asynchrony and versatility

Why can EDA greatly reduce the degree of coupling?

A more pragmatic definition of coupling is: the degree to which a component is affected by other components. Coupling exists in space (components are structurally related) and time (time affects the degree of relationship between components). For the latter, a better example is that one service calls the REST API of other services synchronously. If the called service is down, the service will not be able to continue processing (the response is blocked). If two services must run at the same time, there will be a certain degree of temporal coupling between the two . If two services are highly dependent, it is called a strong coupling, otherwise, it is called a loose coupling.

Conceptual model of coupling

EDA uses two methods to suppress coupling.

To recap, events cannot be communicated, they only happen . The component that initiated the event (via the release record) does not know whether other components exist. Therefore, even if the consumer is unavailable, the producer will not stop working-the broker will temporarily cache the event without back pressure on the producer.
The broker's persistence of event records greatly eliminates the concept of time. A producer might T1 Published an event, a consumer might T2 will read the event incident, T1 and T2 interval between might be in milliseconds (all components of normal) or hour level ( If some consumers are down or busy with other things).

EDA is not a silver bullet, it does not eliminate the concept of coupling (otherwise, the components in the system will no longer work together). Now shift the focus to the broker: in order for producers and consumers to meaningfully decouple, they must rely on a broker. This approach increases the complexity of the system architecture and introduces other points of failure. This is why brokers must be high-performance and fault-tolerant, otherwise, we just replace one set of problems with another.

The way the event is handled

Time processing is usually divided into three common ways. These methods are not mutually exclusive, and they often coexist in a large event-driven system.

Discrete event processing

Used to handle discrete events: such as publishing a post on a social media platform. Discrete event processing is characterized in that the events that appear are usually not related and can be processed independently.

Event stream processing

It is used to process a series of related unbounded event streams. The records of the events are presented in a certain order and carry some information related to the events that occurred. For example, when a business entity undergoes a joint change, the consumer may make the changes in the order specified by the producer and save a copy of the entity in the local database. Due to the need to pay attention to the sequence of event processing, such events cannot be handled discretely. Consumers need to avoid conditional competition, that is, multiple consumer instances may modify a record in the database at the same time, which may cause data inconsistency due to out-of-order updates.

More well-known streaming event platforms, such as Kafka, rely on recorded keys and partitions to preserve the update order. Kafka also guarantees that all changes to an entity will be processed by a certain consumer, avoiding multiple consumers processing events in parallel and causing concurrent competition.

Complex event handling

Complex event processing (CEP) is a model for deriving or identifying complex events from a series of simple events. For example, monitoring the temperature and delay sensors in a building, and inferring whether a fire has occurred, and continuing to track them. A temperature change alone is not enough to trigger an alarm. What is more significant is the cluster event formed by the convergence of temperature peaks and rate of change, which may save lives.

This type of processing is usually more involved, requiring event handlers to keep track of previous events and provide an effective way to request and aggregate.

When to use EDA

The advantages of event-driven architecture can be used in some scenarios:

An opaque consumer ecosystem. In this case, the producer does not understand the consumer. The latter may be a short process, which may come and go in a short period of time!
High fan-out. A scenario where an event may be handled by multiple different consumers.
Complex pattern matching. It is possible to string events together to infer more complex events. ( Such scenarios may require aggregation, that is, the complex event processing described above )
Separation of responsibilities for command query. CQRS is a mode that separates the read and update operations of the data storage area. Implementing CQRS can improve the scalability and flexibility of the application (a trade-off is made in data consistency). This mode is usually associated with EDA.

Benefits of EDA

Caching and fault tolerance. The rate of event consumption may not be synchronized with the producer, and the producer cannot slow down in order to be consistent with the consumer.
Producers and consumers are decoupled to avoid clumsy point-to-point integration. Under EDA, it is easy to add new producers and consumers, and it is also easy to modify the implementation of producers and consumers (provided that the contract/scheme that constrains event records is complied with).
Large-scale expansion. Usually part of the event stream is divided into several unrelated self-streams, and then processed in parallel. With the backlog of events, we can also expand the number of consumers to meet the load demand. Platforms like Kafka will process events strictly in order and allow massively parallel processing across streams.

Disadvantages of EDA

Only asynchronous processing. Although EDA is an effective system decoupling mode, it also limits the application to asynchronous event processing. EDA does not handle interactions like request and response well (the initiator must wait for the response to continue processing).
Introduce additional complexity. Traditional client-server and request-response only involve two parties. After adopting EDA, a third-party broker is introduced as an intermediary between producers and consumers.
The fault is concealed. This is peculiar because it seems to run counter to the nature of decoupling systems. When the system is highly coupled, errors in a system will be passed down quickly and attract our attention. In most scenarios, we need to avoid this situation: when a component fails, try to minimize its impact on other components. The negative effect of fault concealment is that it inadvertently hides problems that should have caught our attention. It can be solved by adding real-time monitoring and logging to each event-driven component, but doing so also brings new complexity.

Points to note

EDA is not a panacea. Like many powerful tools, it can be used incorrectly. The content listed below should not be considered as the shortcomings of EDA, but as a series of pitfalls that developers and architects should pay attention to when designing and implementing event-driven systems.

Complex choreography. Using loosely coupled components, users may be confused. The entire architecture looks like a Rube Goldburg machine ( you can use the following figure to understand Rube Goldburg ), and the entire business logic is also implemented as a series of events (packaged with side effects): An event initiated by one component may trigger another component to initiate another event, and then trigger another component to initiate an event, and so on. This kind of interaction between components can quickly become incomprehensible.
Confusion of commands and events. An event is used to simply describe what happened. It does not specify how to handle the event. A command is a direct command for a specific component. Since both commands and events are certain types of messages, it is very easy to confuse the command and mistake it for an event.

Commands can also be placed under EDA, but they must be distinguished from events. Commands may modify the state of the system and usually require a rollback plan.
Consumers are agnostic. The event should capture the relevant properties in some way, but it does not limit how to handle these events. easy to say, hard to do. Sometimes we may not be able to obtain enough information to limit the content added to the event record (it is impossible to determine whether the information added to the record is ultimately useful).

I think the most important thing is the second point above, to distinguish between commands and events.

to sum up

The microservice architecture pattern is one of the difficult problems involved in building a more maintainable, scalable, and robust software system. From the perspective of problem decomposition, microservices are great, but they also bring a lot of thorny problems, one of which is coupling. Randomly splitting the system into a small number of microservices may put you in a worse situation than at the beginning. There is a term to describe it "distributed all-in-one".

To help solve the confusion and locate the coupling problem, we introduced an event-driven architecture.

EDA is an effective tool that can help reduce the coupling between system components. It is a model that uses producers, consumers, events, and streams to interact. An event represents an action of interest, and any component may publish and consume events asynchronously without needing to perceive the other's existence. EDA allows components to operate and evolve independently. But it is not the silver bullet that solves all problems. EDA is a good choice, and its benefits greatly exceed the cost of adopting it. It can be said that EDA is a necessary element for the successful deployment of microservices.