ServiceComb data consistency solution Saga evolution introduction

This article is transferred from the official blog of the microservice open source project Apache ServiceComb (incubating):

http://servicecomb.incubator.apache.org/cn/docs/saga_pack_design/

In the process of microservice transformation of traditional monolithic applications, most of them will face database splitting, so the data consistency originally guaranteed by the database must also be redesigned and implemented. At this time, a distributed data consistency solution needs to be introduced. Common solutions mainly include 2PC, TCC, event-driven, etc. In the microservice open source project ServiceComb, the use of Saga [1] is proposed and implemented to solve the data consistency problem of microservices. For comparison of different solutions, please refer to "ServiceComb" The data eventual consistency scheme" [2] . Saga is a solution for eventual consistency of data, which allows us to execute all transactions successfully, or in case of any transaction failure, to compensate for successful transactions, and provides the guarantees of ACD in ACID [3] (since transactions are Interleaved execution may see partial results of other transactions, so isolation requirements cannot be met). Therefore, Saga is suitable for the following cross-service transaction scenarios:

Nested calls . For example, when shopping online, it will go through the sub-transactions of order placement, payment service and third-party payment. Among them, the order placement depends on the return status of the payment service, and the payment service also includes a variety of optional payment methods, and depends on the payment service. The result returned by the specific payment method. Through Saga, you can clearly see the relationship between various services in a complete transaction, and can quickly locate the sub-transaction that has problems in the event of an exception.
High concurrency . For example, in the seckill scenario, the seckill can be considered a success after the inventory is successfully deducted and the payment is completed. If the inventory is successfully deducted but the payment fails, compensation will be automatically performed (that is, the inventory will be restored). Since Saga has only two states of commit and compensation, only one call to each sub-transaction is required in a successful scenario, so high performance can be maintained under high concurrency.
Call time is long . If you buy a movie ticket online, there is usually a 15-minute payment time after you have selected your seat. Saga only locks resources for a short period of time in the commit phase of the sub-transaction, and uses a timeout mechanism to ensure that the transaction can be automatically compensated after timeout, that is, if the payment is not successful within the specified time, the locked seat will be automatically released, which greatly simplifies business exceptions. time processing logic.

The evolution of the new version of Saga

New year and new weather, Apache ServiceComb(incubating) Saga [4] (hereinafter referred to as Saga) has evolved. Compared with the previous version [2] , the newly evolved design mainly has the following advantages:

Greatly improve ease of use . Developers only need to use 2-3 annotations (ie enable transaction service: EnableOmega, global transaction flag: SagaStart and sub-transaction flag: Compensable).
Easier to expand . More friendly support for microservice frameworks.
Data consistency is decoupled from business logic . In the evolved design, through the introduction of the service-side omega, the saga coordinator has a more single responsibility (only needs to be responsible for coordinating the integrity of the transaction) and has nothing to do with the specific business, so developers can focus on the development of the specific business.

The evolved architecture of Saga, as shown in the figure below, mainly includes two components, namely alpha and omega, among which:

Alpha acts as a coordinator, mainly responsible for persistent storage of transaction events and coordinating the state of sub-transactions, so that it can be consistent with the state of the global transaction, that is, to ensure that the sub-transactions in the transaction are either fully executed or not at all. implement.
Omega is an agent embedded in the microservice, which is responsible for intercepting network requests and reporting transaction events to alpha, and performing corresponding compensation or retry operations according to the instructions issued by alpha under abnormal conditions.

The inner workings of omega

An omega is an agent embedded in a microservice, responsible for reporting transaction status to alpha and directly passing transaction context information to other omegas. Among them, the transaction context for each service includes:

Global transaction id (Global Tx Id): used to uniquely identify a global transaction, generally generated at the global transaction entry, and transmitted throughout the transaction process.
Local transaction id (Local Tx Id): used to uniquely identify the local transaction, generally generated by the local transaction.
Parent transaction id (Parent Tx Id): used to build the relationship between child transactions, which can be built in the request context.

As shown in the figure below, the processing flow of distributed transactions is very similar to that of zipkin [5] for distributed call chain tracing . On the service provider side, omega will intercept the request and extract the global transaction id in the request information as its own The global transaction id (that is, the Saga event id), and the local transaction id in the request is used as its parent transaction id, and the newly generated id is used as the local transaction id; on the service consumer, omega will intercept the request and add it to it Current global transaction id and local transaction id. Through this cooperative processing of service providers and service consumers, sub-transactions can be connected to form a complete global transaction.

In the preprocessing stage, omega will first send a transaction start event to alpha, and in the post-processing stage, it will send a transaction end event to alpha. Alpha will store persistently after receiving the event. Therefore, each successful subtransaction has a one-to-one corresponding start and end events.

When omega starts, it will register with alpha, so that in the event of an exception or timeout, alpha can send retry or compensation commands and corresponding call parameters to omega through callbacks, thereby ensuring the consistency of global transactions.

Specific processing flow

success scenario

Before the global transaction starts, omega will first send a global transaction start event to alpha, and send a global transaction end event to alpha when all subtransactions are completed. Each sub-transaction will also send a transaction start event to alpha before execution, and after successful execution, a transaction end event will be sent to alpha. Subtransactions are linked together by global transaction ids, but are also differentiated by local transaction ids. Therefore, in a success scenario, every starting event will have a corresponding ending event.

abnormal scene

When an exception is thrown during the execution of a sub-transaction, omega will report an aborted event to alpha, and then alpha will send compensation instructions to other completed sub-transactions of the global transaction to ensure that all sub-transactions under the same global transaction are either successful, Either roll back. Since the participants in the global transaction are not explicitly specified in the transaction, alpha's scanner periodically queries the event table and finds out the global transactions that have completed all compensating subtransactions, and then adds a global transaction end event to these global transactions to guarantee the transaction completeness.

Timeout Scenario

Alpha's scanner will periodically scan the status of the events being processed. If an event is found to have timed out, it will record the corresponding aborted event, and then alpha will send compensation commands to other completed sub-transactions of the global transaction to restore to the one before the transaction started. condition.

how to use?

The use of Saga mainly covers two aspects, the startup of alpha and the use of omega.

alpha launch

The database PostgreSQL needs to be run before the alpha starts:

docker run -d -e "POSTGRES_DB=saga" -e "POSTGRES_USER=saga" -e "POSTGRES_PASSWORD=password" -p 5432:5432 postgres

Copy

After making sure the database is up properly, you can run the alpha:

docker run -d -p 8090:8090 \
  -e "JAVA_OPTS=-Dspring.profiles.active=prd" \
  -e "spring.datasource.url=jdbc:postgresql://{docker.host.address}:5432/saga?useSSL=false" \
  alpha-server:0.1.0

Copy

Use of omega

The use of omega is very simple. Taking a simplified transfer business as an example, the same transfer either succeeds in and out, or fails. Introducing Saga into such a business is just a few simple steps:

import dependencies

<dependency>
  <groupId>org.apache.servicecomb.saga</groupId>
  <artifactId>omega-spring-starter</artifactId>
  <version>0.1.0</version>
</dependency>
<dependency>
  <groupId>org.apache.servicecomb.saga</groupId>
  <artifactId>omega-transport-resttemplate</artifactId>
  <version>0.1.0</version>
</dependency>

Copy

Add Saga's annotations and corresponding compensation methods 2.1 @EnableOmega The annotations added at the application entry to initialize the configuration of omega and establish a connection with alpha.

 @SpringBootApplication
 @EnableOmega
 public class Application {
   public static void main(String[] args) {
     SpringApplication.run(Application.class, args);
   }
 }

Copy

@SagaStart 2.2 Annotations added at the beginning of a global transaction .

 @SagaStart(timeout=10)
 public boolean transferMoney(String from, String to, int amount) {
   transferOut(from, amount);
   transferIn(to, amount);
 }

Copy

2.3 The annotations added to the sub-transaction @Compensable indicate the corresponding compensation method. Among them, the formal parameter list of the compensation method must be consistent with the formal parameter list of the sub-transaction method.

 @Compensable(timeout=5, compensationMethod="cancel")
 public boolean transferOut(String from, int amount) {
   repo.reduceBalanceByUsername(from, amount);
 }
    
 public boolean cancel(String from, int amount) {
   repo.addBalanceByUsername(from, amount);
 }

Copy

2.4 Repeat step 2.3 for the transfer-in service.

2.5 Add a configuration item to the application.yaml of each service to indicate the service information and alpha address information:

 spring:
   application:
     name: {application.name}
 alpha:
   cluster:
     address: {alpha.cluster.addresses}

Copy

At present, there are still many interesting and challenging topics in the implementation of Saga, such as the realization of alpha coordination scheduling, the realization of idempotency and the realization of automatic compensation, etc. We welcome people with lofty ideals to work with us to solve the problem of data consistency. Contribute to the improvement of the microservice ecosystem.