How to ensure data consistency when placing an order?

insert image description here

Hello everyone, I am Nezha.

In the previous articles, we mentioned the Redis implementation rankings and the Redis data caching strategy , which gave us a better understanding of Redis. Today we will continue to learn more and learn how Redis ensures data consistency when placing orders?

For example, under high concurrent access, there may be multiple requests to read the same cached data at the same time, and then perform write operations, which is prone to data competition. At the same time, read and write operations are not atomic operations. When reading data, the cache may have been updated by other requests, resulting in data inconsistency.

In order to solve the data consistency problem of Redis cache, we need to do the following two things:

Ensure that all requests read the latest data ;
All update operations are guaranteed to be mutually exclusive and performed in the requested order .

insert image description here

In an online mall system, we face an important issue: how to ensure data consistency during the order payment process, and how to optimize the performance of payment operations.

1. Order payment requirements

After the user places an order, the order payment operation needs to be performed to ensure the consistency of payment and order status.

2. Data consistency requirements

After successful payment, the order status must be updated to Paid to maintain data consistency.

3. High concurrent payment

In the case of high concurrency, it is necessary to ensure the performance and data consistency of order payment.

In order to solve the above problems, we can use the transaction and pipeline mechanisms provided by Redis.

1. Redis transaction

1. What is a Redis transaction

insert image description here

In Redis, a transaction is a collection of commands that can be executed in a separate process to ensure the atomicity, consistency, isolation, and durability of these commands.

(1) Transaction overview

Redis transactions are managed by the following four key commands:

Order	describe
MULTI	Start a transaction and mark the beginning of a transaction block.
EXEC	Execute all commands in the transaction.
DISCARD	Cancel the transaction and discard all enqueued commands.
WATCH	Monitors one or more keys for optimistic locking.

(2) Transaction characteristics of Redis

Redis transactions have the following key characteristics:

transaction characteristics	describe
atomicity	All commands in the transaction are either executed or not executed. This ensures that during transaction execution, it does not happen that some commands succeed and some fail.
consistency	The commands in the transaction will be executed in the order they are added and will not be interrupted by commands from other clients. This ensures that operations within a transaction are executed in the expected order and will not be affected by concurrent operations.
Isolation	During transaction execution, the transaction is isolated and will not be affected by other transactions. Even if there are other concurrent transactions executing, the operations in the transaction will not be seen by other transactions until the transaction is executed and committed.
persistence	Modifications to the database will be persisted to disk after the transaction is executed. This ensures that operations within a transaction are not lost due to system failure, thus ensuring data durability.

The above are the basic concepts and characteristics of Redis transactions, which ensure that transactions executed in Redis are reliable and consistent sets of operations.

insert image description here

The above graphic represents the interrelationship between the key features of Redis transactions. These features support each other and jointly ensure the reliability and consistency of Redis transactions.

Atomicity guarantees that operations in a transaction either all succeed or all fail.
Consistency ensures that operations in a transaction are executed in a specific order without being interfered by other operations.
Isolation ensures that transactions are isolated from other transactions during execution and do not interfere with each other.
Durability ensures that modifications after transaction execution will be persisted and will not be lost due to system failure. Together, these features form the basis for the reliability and stability of Redis transactions.

2. Use Redis transaction

(1) Start and submit transactions

In Redis, you need to follow the following steps to use transactions:

Use MULTIthe command to start the transaction.
Execute the commands that need to be executed within the transaction.
Use EXECthe command to commit the transaction and execute all commands in the transaction.

Here are the detailed steps using a Java code example:

// 创建与Redis服务器的连接
Jedis jedis = new Jedis("localhost", 6379);

// 开启事务
Transaction transaction = jedis.multi();

// 执行事务中的命令
transaction.set("key1", "value1");
transaction.set("key2", "value2");

// 提交事务并获取执行结果
List<Object> results = transaction.exec();

In the above example, the transaction.set("key1", "value1")and transaction.set("key2", "value2")commands will be added to the transaction queue, and when transaction.exec()called, all commands in the transaction will be executed together. If an error occurs between and , the transaction will be canceled and the command will not be executed MULTI.EXEC

(2) Transaction command

Within a transaction, you can use regular Redis commands such as SET, GET, HSET, ZADDetc. These commands are added to the transaction queue until EXECthe command is executed.

(3) Transaction example

The following is an example using Java code to demonstrate executing common Redis commands within a transaction:

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Transaction;

public class RedisTransactionCommandsExample {
    
    
    public static void main(String[] args) {
    
    
        Jedis jedis = new Jedis("localhost", 6379);

        // 开启事务
        Transaction transaction = jedis.multi();

        // 执行事务中的命令
        transaction.set("name", "Alice");
        transaction.hset("user:1", "name", "Bob");
        transaction.zadd("scores", 100, "Alice");
        transaction.zadd("scores", 200, "Bob");

        // 提交事务并获取执行结果
        List<Object> results = transaction.exec();

        // 打印执行结果
        for (Object result : results) {
    
    
            System.out.println("Result: " + result);
        }

        // 关闭连接
        jedis.close();
    }
}

In the above example, the SET, HSETand ZADDcommands are used, and these commands are added to the transaction queue. When executed transaction.exec(), all commands in the transaction are executed together. The examples here are simple demonstrations, you can add more commands to build more complex transactions as needed.

2. Redis Pipeline

1. What is a Redis pipeline

Redis pipeline (Pipeline) is a technology that optimizes Redis operations. It allows multiple commands to be sent to the Redis server in a single communication, thereby significantly reducing communication overhead and improving performance.

Pipes can send multiple commands to the server at once without waiting for the response of each command, which allows Redis to handle batch operations and large-scale data reading and writing more efficiently.

The following figure shows how the Redis pipeline works:

insert image description here

In the figure above, the client (Client) sends multiple commands to the Redis server (Server), and each command is represented by Command 1, Command 2etc. These commands are sent to the server all at once without waiting for a response for each command. After executing all commands, the server responds to the client with the results at once. It also explains how the Redis pipeline works: by packaging multiple commands into one communication, the communication overhead of each command is reduced and the performance of the system is improved.

When using a Redis pipeline, the client creates a pipeline object, adds multiple commands to the pipeline, and then executes the commands in the pipeline at once. Finally, the client can collect the execution results of all commands.

(1) Pipeline overview

In Redis, pipelines are managed through the following commands:

Order	describe
`PIPELINE`	Enable pipeline mode to send multiple commands at once.
`MULTI`	Turn on transaction mode, which is used to execute a series of commands in the pipeline.
`EXEC`	Commit the transaction in the pipeline, execute it and return the result.

Using pipes, you can send multiple commands to the server at once, and then obtain the execution results of all commands through one communication, thereby reducing the communication overhead of each command and improving the performance of the system.

(2) Pipeline characteristics of Redis

Using Redis pipelines provides the following advantages:

Reduce communication overhead: In ordinary command transmission, each command requires back-and-forth network communication, while pipelines can package multiple commands and send them to the server at once, thus greatly reducing communication overhead. This is especially important for scenarios with high network latency, effectively improving performance.
Improved throughput: Pipes allow multiple commands to be executed in one communication, thereby processing more commands per unit of time. This can effectively improve Redis's throughput and response capabilities for scenarios that require processing a large number of commands, such as batch data processing, concurrent request processing, etc.

2. Use the Redis pipeline

(1) Pipeline commands

The following is a practical case showing how to use Redis pipeline to execute multiple commands and improve performance:

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import java.util.List;

public class RedisPipelineExample {
    
    
    public static void main(String[] args) {
    
    
        Jedis jedis = new Jedis("localhost", 6379);

        // 创建管道
        Pipeline pipeline = jedis.pipelined();

        // 向管道中添加命令
        for (int i = 0; i < 10000; i++) {
    
    
            pipeline.set("key" + i, "value" + i);
        }

        // 执行管道中的命令
        List<Object> results = pipeline.syncAndReturnAll();

        // 关闭连接
        jedis.close();
    }
}

In the above case, a loop is used to add 10,000 commands to the pipeline SET. By using pipes, all commands can be sent to the server in one communication rather than one at a time, thus reducing communication overhead and improving performance.

(2) Pipeline optimization performance

Using Redis pipelines can improve performance, especially if you need to batch process multiple commands. The principle of the pipeline is to send multiple commands to the server at once, and then obtain the results at once, which reduces the number of communication round-trips and thus significantly improves throughput.

However, there are a few things to note:

Pipelines do not support transactions and cannot guarantee atomic execution of multiple commands.
When using pipelines, the execution order of commands may be inconsistent with the order of addition, which needs to be considered based on business needs.
Pipelines do not bring performance improvements in all scenarios and need to be evaluated based on actual conditions.

Through reasonable use of pipelines, the advantages of Redis in high-performance data processing can be maximized.

insert image description here

3. Transactions vs. Pipelines: When to use which?

1. Applicable scenarios for transactions

Transactions can guarantee atomicity and consistency in certain scenarios, and are especially suitable for business operations with strong consistency requirements, such as payment operations.

(1) Strong consistency operation

Transactions are a mechanism suitable for operations that require strong consistency. When multiple commands need to be executed atomically in a sequence of operations, transactions can ensure that either all of the commands are executed or none of them are executed to maintain data consistency.

In the following example, a bank transfer operation is simulated, where the balance of one account needs to be deducted and the balance of another account needs to be increased at the same time:

Jedis jedis = new Jedis("localhost", 6379);

// 开启事务
Transaction transaction = jedis.multi();

// 扣减账户1余额
transaction.decrBy("account1", 100);

// 增加账户2余额
transaction.incrBy("account2", 100);

// 提交事务并获取执行结果
List<Object> results = transaction.exec();

// 关闭连接
jedis.close();

(2) High atomicity requirements

Transactions are a better choice when the business requires that multiple operations either all succeed or all fail. Transactions ensure that a series of commands in a transaction are executed atomically, thereby maintaining data consistency.

2. Applicable scenarios of pipelines

Pipelines are suitable for scenarios that require batch operations and high throughput requirements. By sending multiple commands to the server at once, communication overhead can be reduced and performance improved.

(1) Batch operation

Use pipelines to perform batch operations efficiently. For example, when you need to add a large amount of data to a database, using pipelines can reduce the communication cost of each command, thereby greatly improving the efficiency of the operation.

The following example demonstrates how to use pipes for batch setup operations:

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Pipeline;
import java.util.List;

public class RedisPipelineBatchExample {
    
    
    public static void main(String[] args) {
    
    
        Jedis jedis = new Jedis("localhost", 6379);
        Pipeline pipeline = jedis.pipelined();

        // 向管道中添加一批设置操作
        for (int i = 0; i < 1000; i++) {
    
    
            pipeline.set("key" + i, "value" + i);
        }

        // 执行管道中的命令
        List<Object> results = pipeline.syncAndReturnAll();

        // 关闭连接
        jedis.close();
    }
}

(2) High throughput requirements

In scenarios where high throughput is required, pipelines can significantly improve performance. When multiple commands need to be executed in a short period of time, pipes can be used to package and send these commands, reducing the number of communication round-trips.

When using pipelines for large-scale data processing, it can especially improve the system's processing capabilities under high load conditions.

insert image description here

4. Case study: ensuring data consistency and performance optimization for order payment

1. Scene description

In an online mall system, we face an important issue: how to ensure data consistency during the order payment process, and how to optimize the performance of payment operations.

(1) Order payment requirements

After the user places an order, the order payment operation needs to be performed to ensure the consistency of payment and order status.

(2) Data consistency requirements

After successful payment, the order status must be updated to Paid to maintain data consistency.

(3) High concurrent payment

In the case of high concurrency, it is necessary to ensure the performance and data consistency of order payment.

2. Use Redis transactions to solve data consistency problems

(1) Transaction realizes order payment

Jedis jedis = new Jedis("localhost", 6379);
Transaction transaction = jedis.multi();

// 扣除用户余额
transaction.decrBy("user:balance:1", orderAmount);

// 更新订单状态为已支付
transaction.hset("order:1", "status", "paid");

List<Object> results = transaction.exec();

In the above example, a Redis transaction is used to ensure that the deduction of the user's balance and the update of the order status occur at the same time in an operation sequence. If any operation in the transaction fails, the entire transaction will be rolled back, ensuring data consistency.

(2) Transaction consistency guarantee

Using transactions can ensure the consistency of user balance and order status, either succeeding at the same time or failing at the same time. This ensures the correctness of payment and order status and avoids potential data inconsistencies.

3. Use Redis pipeline to optimize payment performance

(1) Pipeline batch payment

Jedis jedis = new Jedis("localhost", 6379);
Pipeline pipeline = jedis.pipelined();

for (Order order : orders) {
    
    
    pipeline.decrBy("user:balance:" + order.getUserId(), order.getAmount());
    pipeline.hset("order:" + order.getId(), "status", "paid");
}

List<Object> results = pipeline.syncAndReturnAll();

In this example, a Redis pipeline is used to batch process payments for multiple orders. By sending multiple commands to the server at once, communication overhead can be reduced, significantly improving the performance of payment operations.

(2) Pipeline performance improvement

By using pipelines, multiple payment operations can be packaged in one communication, reducing the number of communication round-trips and thus improving payment performance.

Especially in high-concurrency payment scenarios, pipelines can significantly reduce server load and improve system responsiveness.

insert image description here

5. Restrictions and precautions on transactions and pipelines

1. Transaction restrictions

You need to pay attention to the following restrictions when using transactions, including the use of WATCH commands and optimistic locks .

(1) WATCH command

Use the WATCH command in a transaction to monitor one or more keys. If the monitored key is modified by other clients during the execution of the transaction, the transaction will be interrupted. This is to ensure transaction consistency and avoid race conditions.

Positive example:

Jedis jedis = new Jedis("localhost", 6379);
Transaction transaction = jedis.multi();

// 监视键"balance"
transaction.watch("balance");

// ... 在此期间可能有其他客户端修改了"balance"键的值 ...

// 执行事务
List<Object> results = transaction.exec();

counterargument:

Jedis jedis = new Jedis("localhost", 6379);
Transaction transaction = jedis.multi();

// 监视键"balance"
transaction.watch("balance");

// ... 在此期间其他客户端修改了"balance"键的值 ...

// 尝试执行事务，但由于"balance"键被修改，事务会被中断
List<Object> results = transaction.exec();

(2) Optimistic lock

When dealing with concurrent updates, optimistic locking can be used. By using mechanisms such as version numbers or timestamps, check whether the data has been modified by other clients before executing the command to avoid concurrency conflicts.

Positive example:

Jedis jedis = new Jedis("localhost", 6379);

// 获取当前版本号
long currentVersion = Long.parseLong(jedis.get("version"));

// 更新数据前检查版本号
if (currentVersion == Long.parseLong(jedis.get("version"))) {
    
    
    Transaction transaction = jedis.multi();
    transaction.set("data", "new value");
    transaction.incr("version");
    List<Object> results = transaction.exec();
} else {
    
    
    // 数据已被其他客户端修改，需要处理冲突
}

2. Precautions for pipelines

There are a few things to note when using pipes, including pipe seriality and careful use.

(1) Does not support transactions

Pipes do not support transactions, so transaction atomicity and consistency cannot be achieved through pipes. If transaction support is required, Redis' transaction mechanism should be used.

(2) Use pipelines with caution

Although pipelines can improve performance, they do not bring performance improvements in all scenarios. In some cases, due to the serial nature of the pipeline, some commands may block the execution of other commands, which in turn reduces performance.

Positive example:

Jedis jedis = new Jedis("localhost", 6379);
Pipeline pipeline = jedis.pipelined();

for (int i = 0; i < 1000; i++) {
    
    
    pipeline.set("key" + i, "value" + i);
}

// 执行管道中的命令并获取结果
List<Object> results = pipeline.syncAndReturnAll();

counterargument:

Jedis jedis = new Jedis("localhost", 6379);
Pipeline pipeline = jedis.pipelined();

for (int i = 0; i < 1000; i++) {
    
    
    // 注意：此处执行了耗时的命令，可能阻塞其他命令的执行
    pipeline.get("key" + i);
}

// 执行管道中的命令并获取结果
List<Object> results = pipeline.syncAndReturnAll();

6. Summary

This blog provides an in-depth exploration of the transaction and pipeline mechanisms in Redis and their application in ensuring data consistency and optimizing performance.

Through detailed explanations and code examples, we understand the basic concepts, features, usage and applicable scenarios of transactions and pipelines. The following is a summary of the main content of this blog:

In the Redis transaction section, we learned about the concepts and characteristics of transactions. Transactions ensure the atomicity, consistency, isolation, and durability of a series of commands .

Through the MULTI, EXEC, DISCARD and WATCH commands, we can manage the start, commit, rollback and monitor key changes of transactions. Transactions are suitable for operations that need to ensure atomicity and consistency, especially in scenarios with strong consistency requirements.

In the Redis pipeline section, we took an in-depth look at the concepts and benefits of pipelines. Pipes allow multiple commands to be sent to the server at once, reducing communication overhead and improving performance.

Through the PIPELINE, MULTI, and EXEC commands, we can create pipelines, add commands, and execute commands in the pipeline. Pipelines are suitable for batch operations and scenarios with high throughput requirements, and can significantly improve the performance of Redis.

In the Transactions vs. Pipelines: When to Use Which section, we compared the applicable scenarios of transactions and pipelines.

Transactions are suitable for scenarios that ensure strong consistency operations and high atomicity requirements;
Pipelines are suitable for batch operations and high-throughput scenarios.

Through examples, we illustrate how to choose the appropriate mechanism to meet consistency and performance requirements based on business needs.

In the case study: ensuring data consistency and performance optimization for order payment , we applied previous knowledge to solve a practical problem. We showed how transactions can be used to ensure data consistency for order payments, while pipelines can be used to optimize the performance of payment operations. This case fully reflects the application of transactions and pipelines in actual business.

In the Limitations and Considerations for Transactions and Pipelines section, we point out some limitations and considerations for transactions and pipelines. Transactions are limited by WATCH commands and optimistic locking, while pipelines do not support transactions and need to be used with careful consideration of the performance impact.

Through this blog, we have explored the transaction and pipeline mechanisms in Redis in detail, and learned how they ensure data consistency and optimize performance in practical applications. Whether you emphasize consistency or pursue performance, you can choose the appropriate mechanism based on business needs to achieve the best results.