Let me talk about the conclusion first:

The transaction mode of Redis has the following characteristics:

Guaranteed isolation;
Persistence cannot be guaranteed;
It has certain atomicity, but does not support rollback;
There are differences in the concept of consistency. Assuming that the core of consistency is the semantics of constraints, Redis transactions can guarantee consistency.

However, Lua scripts are more practical. It is another form of transaction. It has certain atomicity, but when the script reports an error, the transaction will not be rolled back. Lua scripts can guarantee isolation, and can perfectly support the subsequent steps to depend on the results of the previous steps .

The Lua script mode is almost everywhere, such as distributed locks, delay queues, red envelope grabbing and other scenarios.

1 Transaction Principle

Redis transactions include the following commands:

serial number	command and description
1	MULTI marks the beginning of a transaction block.
2	EXEC executes all commands within a transaction block.
3	DISCARD cancels the transaction, abandoning the execution of all commands within the transaction block.
4	WATCH key [key ...] monitors one (or more) keys, and if the key (or keys) is changed by other commands before the transaction is executed, the transaction will be interrupted.
5	UNWATCH cancels the monitoring of all keys by the WATCH command.

A transaction consists of three phases:

The transaction is opened, using MULTI, this command marks the client executing the command to switch from the non-transactional state to the transactional state;
The command is enqueued. After MULTI starts the transaction, the client's command will not be executed immediately, but put into a transaction queue;
Execute the transaction or discard it. If an EXEC command is received, the command in the transaction queue will be executed, and if it is DISCARD, the transaction will be discarded.

An example of a transaction is shown below.

 redis> MULTI 
 OK
 redis> SET msg "hello world"
 QUEUED
 redis> GET msg
 QUEUED
 redis> EXEC
 1) OK
 1) hello world

Have a question here? Can the Redis key be modified when opening a transaction?

Before the transaction executes the EXEC command, the Redis key can still be modified .

Before the transaction is started, we can use the watch command to monitor the Redis key. Before the transaction execution, we modify the key value, the transaction execution fails and returns nil .

Through the above example, the watch command can achieve an effect similar to optimistic locking .

2 ACID for transactions

2.1 Atomicity

Atomicity means that all operations in a transaction are either completed or not completed, and will not end in a certain link in the middle. If an error occurs during the execution of the transaction, it will be rolled back to the state before the transaction started, as if the transaction had never been executed.

First example:

Before executing the EXEC command, the operation command sent by the client is wrong, such as: syntax error or non-existing command is used.

 redis> MULTI
 OK
 redis> SET msg "other msg"
 QUEUED
 redis> wrongcommand  ### 故意写错误的命令
 (error) ERR unknown command 'wrongcommand' 
 redis> EXEC
 (error) EXECABORT Transaction discarded because of previous errors.
 redis> GET msg
 "hello world"

In this example, we use a non-existent command, which causes enqueue failure and the entire transaction will not be executed.

Second example:

When the transaction operation is enqueued, the data types of the command and the operation do not match, and the enqueue is normal, but the execution of the EXEC command is abnormal.

 redis> MULTI  
 OK
 redis> SET msg "other msg"
 QUEUED
 redis> SET mystring "I am a string"
 QUEUED
 redis> HMSET mystring name  "test"
 QUEUED
 redis> SET msg "after"
 QUEUED
 redis> EXEC
 1) OK
 2) OK
 3) (error) WRONGTYPE Operation against a key holding the wrong kind of value
 4) OK
 redis> GET msg
 "after"

In this example, if an error occurs when Redis executes the EXEC command, Redis will not terminate the execution of other commands, and the transaction will not be rolled back because a command fails to execute.

In summary, my understanding of the atomicity of Redis transactions is as follows:

When an error is reported when the command enters the queue, transaction execution will be abandoned to ensure atomicity;
It is normal when the command is enqueued, but an error is reported after executing the EXEC command, and the atomicity is not guaranteed;

That is to say: Redis transactions are atomic only under certain conditions .

2.2 Isolation

The isolation of the database refers to the ability of the database to allow multiple concurrent transactions to read, write and modify its data at the same time. The isolation can prevent data inconsistency caused by cross-execution when multiple transactions are executed concurrently.

Transaction isolation is divided into different levels, namely:

Uncommitted read (read uncommitted)
Commit to read (read committed)
Repeatable read (repeatable read)
Serializable

First of all, it needs to be clear: Redis does not have the concept of transaction isolation level. Here we discuss that the isolation of Redis refers to whether transactions can not interfere with each other in a concurrent scenario .

We can divide transaction execution into two phases, before the execution of the EXEC command and after the execution of the EXEC command , and discuss them separately.

Before the EXEC command is executed

In the section on transaction principles, we found that the Redis key can still be modified before the transaction is executed. At this point, you can use the WATCH mechanism to achieve the effect of optimistic locking.

After the EXEC command is executed

Because Redis is a single-threaded execution command, after the EXEC command is executed, Redis will ensure that all commands in the command queue are executed. In this way, the isolation of transactions can be guaranteed.

2.3 Persistence

The persistence of the database means that after the transaction processing is completed, the modification to the data is permanent, even if the system fails, it will not be lost.

Whether Redis data is persisted depends on the persistence configuration mode of Redis.

Without configuring RDB or AOF, the durability of transactions cannot be guaranteed;
Using the RDB mode, after a transaction is executed and before the next RDB snapshot is executed, if an instance downtime occurs, the durability of the transaction cannot be guaranteed;
The AOF mode is used; the three configuration options no and everysec of the AOF mode will cause data loss. always can guarantee the durability of transactions, but because of its poor performance, it is generally not recommended in production environments.

In summary, the durability of redis transactions cannot be guaranteed .

2.4 Consistency

The concept of consistency has always been very confusing. In the information I searched, there are two different definitions.

Wikipedia

Let's first look at the definition of consistency on Wikipedia:

Consistency ensures that a transaction can only bring the database from one valid state to another, maintaining database invariants: any data written to the database must be valid according to all defined rules, including constraints, cascades, triggers, and any combination thereof. This prevents database corruption by an illegal transaction, but does not guarantee that a transaction is correct. Referential integrity guarantees the primary key – foreign key relationship.

In this text, the core of consistency is " constraint ", " any data written to the database must be valid according to all defined rules ".

How to understand constraints? Here is a quote from Han Fusheng, a research and development expert of Ant Financial Services OceanBase, who knows how to understand the internal consistency and external consistency of the database :

"Constraints" are told to the database by the user of the database that the user requires the data to meet certain constraints of one kind or another. When the data is modified, the database will check whether the data still meets the constraint conditions. If the constraint conditions are no longer satisfied, the modification operation will not occur.

The two most common types of constraints in relational databases are "unique constraints" and "integrity constraints". Both the primary key and the unique key defined in the table ensure that the specified data items will never be duplicated, and the referential integrity defined between tables It also ensures the consistency of the same attribute in different tables.

"Consistency in ACID" is so easy to use that it has melted into the blood of most users. Users will consciously add the required constraints when designing tables, and the database will strictly enforce this constraint. condition.

Therefore, the consistency of the transaction is related to the pre-defined constraints, and ensuring the constraints guarantees the consistency .

Let's take a closer look at this sentence: This prevents database corruption by an illegal transaction, but does not guarantee that a transaction is correct .

It may be a bit vague for everyone to write here, let's take a classic transfer case.

We start a transaction. The initial balances on the accounts of Zhang San and Li Si are both 1,000 yuan, and there are no constraints on the balance field. Zhang San transfers 1,200 yuan to Li Si. Zhang San's balance is updated to -200, and Li Si's balance is updated to 2200.

From the perspective of the application level, this transaction is obviously illegal, because in real scenarios, the user balance cannot be less than 0, but it fully complies with the constraints of the database, so from the perspective of the database level, this transaction still guarantees consistency.

The transactional consistency of Redis means that the Redis transaction conforms to the constraints of the database during execution and does not contain illegal or invalid error data.

We discuss three exception scenarios:

Before executing the EXEC command, the operation command sent by the client is wrong, the transaction is terminated, and the data remains consistent;
After executing the EXEC command, if the data type of the command and the operation do not match, an error will be reported for the wrong command, but the transaction will not be terminated because of the wrong command, but will continue to execute. The correct command is executed normally, and the wrong command reports an error. From this point of view, the data can also maintain consistency;
During the execution of the transaction, the Redis service is down. Here you need to consider the persistence mode of the service configuration.
- Non-persistent memory mode: After the service is restarted, the database does not maintain data, so the data is consistent;
- RDB / AOF mode: After the service is restarted, Redis restores the data through the RDB / AOF file, and the database will be restored to a consistent state.

To sum up, under the semantics that the core of consistency is constraint, Redis transactions can guarantee consistency .

"Designing Data-Intensive Applications"

This book is a god book for getting started with distributed systems. There is an explanation of ACID in the chapter on transactions:

Atomicity, isolation, and durability are properties of the database,whereas consistency (in the ACID sense) is a property of the application. The application may rely on the database’s atomicity and isolation properties in order to achieve consistency, but it’s not up to the database alone. Thus, the letter C doesn’t really belong in ACID.

Atomicity, isolation, and durability are properties of the database, while consistency (in the ACID sense) is a property of the application. Applications may rely on the atomicity and isolation properties of the database for consistency, but it doesn't depend solely on the database. Therefore, the letter C is not ACID.

Many times, the consistency we have been struggling with actually refers to the consistency of the real world , and the consistency of the real world is the ultimate goal of transactions.

In order to achieve consistency in the real world, the following points need to be met:

Guarantee atomicity, persistence and isolation. If these characteristics cannot be guaranteed, then the consistency of transactions cannot be guaranteed;
The constraints of the database itself, such as the string length cannot exceed the column limit or unique constraints;
The business level also needs to be guaranteed.

2.5 Transaction Characteristics

We usually call Redis an in-memory database, which is different from traditional relational databases. In order to provide higher performance and faster writing speed, some balance has been made at the design and implementation levels, and it cannot fully support ACID of transactions.

Redis transactions have the following characteristics:

Guaranteed isolation;
Persistence cannot be guaranteed;
It has certain atomicity, but does not support rollback;
There are differences in the concept of consistency. Assuming that the core of consistency is the semantics of constraints, Redis transactions can guarantee consistency.

From an engineering point of view, assuming that each step in the transaction operation needs to rely on the result returned by the previous step, you need to implement optimistic locking through watch.

3 Lua scripts

3.1 Introduction

Lua is written in standard C, the code is concise and beautiful, and can be compiled and run on almost all operating systems and platforms. Lua scripts can be easily called by C/C++ code, and can also call C/C++ functions in turn, which makes Lua widely used in applications.

Lua scripts shine in the field of games. The well-known "Westward Journey II" and "World of Warcraft" all use Lua scripts extensively. API gateways that Java back-end engineers have come into contact with, such as Openresty and Kong , can see Lua scripts.

Starting from Redis version 2.6.0, Redis has a built-in Lua interpreter that can run Lua scripts in Redis.

Benefits of using Lua scripts:

Reduce network overhead. Send multiple requests at once in the form of scripts to reduce network delay.
atomic operation. Redis will execute the entire script as a whole, and will not be inserted by other commands in the middle.
reuse. The script sent by the client will be permanently stored in Redis, and other clients can reuse this script without using code to complete the same logic.

Common commands for Redis Lua scripts:

serial number	command and description
1	EVAL script numkeys key [key ...] arg [arg ...] Execute Lua script.
2	EVALSHA sha1 numkeys key [key ...] arg [arg ...] execute Lua script。
3	SCRIPT EXISTS script [script ...] Check whether the specified script has been saved in the cache.
4	SCRIPT FLUSH removes all scripts from the script cache.
5	SCRIPT KILL kills the currently running Lua script.
6	SCRIPT LOAD script adds the script script to the script cache, but does not execute the script immediately.

3.2 EVAL command

Command format:

 EVAL script numkeys key [key ...] arg [arg ...]

illustrate:

script is the first parameter, which is a Lua 5.1 script;
The second parameter numkeys specifies how many keys there are in subsequent parameters;
key [key ...]KEYS[1], is the key to be operated, you can specify more than one, and get it through the Lua script KEYS[2] ;
arg [arg ...]ARGV[1], parameter, obtained through , in the Lua script ARGV[2] .

Simple example:

 redis> eval "return ARGV[1]" 0 100 
 "100"
 redis> eval "return {ARGV[1],ARGV[2]}" 0 100 101
 1) "100"
 2) "101"
 redis> eval "return {KEYS[1],KEYS[2],ARGV[1]}" 2 key1 key2 first second
 1) "key1"
 2) "key2"
 3) "first"
 4) "second"

The following demonstrates how Lua calls the Redis command, and redis.call() executes the Redis command through it.

 redis> set mystring 'hello world'
 OK
 redis> get mystring
 "hello world"
 redis> EVAL "return redis.call('GET',KEYS[1])" 1 mystring
 "hello world"
 redis> EVAL "return redis.call('GET','mystring')" 0
 "hello world"

3.3 EVALSHA command

Each request using the EVAL command needs to transmit the Lua script. If the Lua script is too long, it will not only consume network bandwidth, but also have a certain impact on the performance of Redis.

The idea is to cache the Lua script first, and return the sha1 summary of the Lua script to the client. The client stores the sha1 digest of the script, and executes the EVALSHA command for each request.

The basic syntax of the EVALSHA command is as follows:

 redis> EVALSHA sha1 numkeys key [key ...] arg [arg ...]

Examples are as follows:

 redis> SCRIPT LOAD "return 'hello world'"
 "5332031c6b470dc5a0dd9b4bf2030dea6d65de91"
 redis> EVALSHA 5332031c6b470dc5a0dd9b4bf2030dea6d65de91 0
 "hello world"

4 Transactions VS Lua scripts

By definition, a script in Redis is itself a transaction , so anything that can be done in a transaction can also be done in a script. And in general, using scripts is simpler and faster .

Because the script function was introduced in Redis 2.6, and the transaction function existed earlier, so Redis will have two methods of processing transactions at the same time.

However , we don't intend to remove the transaction function anytime soon , because transactions provide a way to avoid race conditions without using scripts, and the implementation of transactions themselves is not complicated.

-- https://redis.io/

Lua script is another form of transaction, which has a certain degree of atomicity, but when the script reports an error, the transaction will not be rolled back. Lua scripts can guarantee isolation, and can perfectly support the subsequent steps to depend on the results of the previous steps .