[Redis] Redis learning tutorial (5) Redis transactions, persistence, and clustering

1. Affairs

1.1 Overview of the transaction

Redis transactions are implemented through MULTI, EXEC, DISCARD, several commandsWATCH

• MULTI: open transaction
• EXEC: commit transaction
• DISCARD: Discard the transaction
• WATCH: Provides check-and-set (CAS) behavior for Redis transactions

Redis transactions can execute multiple commands at one time. Redis transactions have the following characteristics:

• A transaction is a single isolated operation: all commands in the transaction are serialized and executed sequentially. During the execution of the transaction, it will not be interrupted by command requests sent by other clients.
• A transaction is an atomic operation: either all commands in the transaction are executed, or none of them are executed.

The transaction operation is as follows:

1.2 Transaction error occurs

Reids transaction errors are divided into two situations:

• An error occurred before transaction commit: An error occurred during sending the command
• Error occurred after transaction commit: An error occurred during the execution of the command

①: An error occurred before the transaction was submitted.

Here the is written incorrectly on incr 命令purpose. From the result, we can see that this incr 命令is not enqueued, and the transaction execution fails, and both num1 and num2 have no value.

②: An error occurred after the transaction was submitted

In the above transaction command, I num1set an a for , then executed the auto-increment command, and finally obtained the value of num1.

We found that an error occurred during the execution of the second command, but the entire transaction was still submitted successfully.

It can be concluded from the above phenomenon that Redis transactions do not support rollback operations. If supported, the entire transaction's commands should not be executed.

1.3 Why does Redis not support transaction rollback?

Here are the advantages of this approach:

• Redis commands will only fail due to incorrect syntax (and these problems cannot be discovered during enqueuing), or the command is used on the wrong type of key: this means that, from a practical point of view, the command fails are caused by programming errors that should be discovered during development and should not appear in production.
• Because rollback support is not required, the internals of Redis can be kept simple and fast.

There is a view that the way Redis handles transactions will cause bugs. However, it should be noted that under normal circumstances, rollback cannot solve the problems caused by programming errors. For example, if you originally wanted to add 1 to the value of a key through the incr command, but accidentally added 2, or executed incr on the wrong type of key, rollback has no way to handle these situations.

1.4 Abandon transaction

When the command is executed discard, the transaction will be aborted, the transaction queue will be emptied, and the client will exit from the transaction state

1.5 WATCH command

WATCH mechanism:Use WATCH to monitor one or more keys and track the value modification of the key. If the value of any key is modified before the transaction EXEC is executed, the entire transaction will be cancelled.. EXEC returns a prompt message indicating that the transaction has failed.

WATCH monitors a key with an expiration time, so even if the key expires, the transaction can still be executed normally

1.6 Cancel WATCH

• The WATCH command can be called multiple times. The key is monitored from the moment WATCH is executed until EXEC is called. Regardless of whether the transaction executes successfully or not, monitoring of all keys will be cancelled.
• When a client disconnects, that client's monitoring of keys is also canceled
• The UNWATCH command can manually unwatch all keys

1.7 Redis scripts and transactions

By definition, a script in Redis itself is a transaction, so anything that can be done in a transaction can also be done in a script. And in general, it's easier and faster to use scripts

2. Sustainability

2.1 Why persistence is needed

We know that Redis is an in-memory database that focuses on high performance and fast speed. Compared with Redis, MySQL's data is saved on the hard disk slowly, but its stability is good.

If you think about it, Redis data is stored in memory. Once the server goes down, all the data will be lost, which will cause a big problem. So in order to make up for this shortcoming, RedisProvides a data persistence mechanism to ensure that data is not lost even if the server goes down.

2.2 Introduction to persistence

Redis provides two persistence mechanisms, RDB and AOF, which are suitable for different scenarios:

• RDB persistence method : Ability to snapshot and store your data at specified time intervals
• AOF persistence method : Record every operation written to the server, and when the server restarts, these commands will be re-executed to restore the original data. The AOF command uses the Redis protocol to append and save each write operation to the end of the file. Redis can also rewrite the AOF file in the background, so that the size of the AOF file will not be too large

2.2.1 RDB

RDB persistence works by taking snapshots of data at specified intervals . For example: to persist data at 8 o'clock, then Redis will fork a child process to persist the data in memory at 8 o'clock to disk.

There are several ways to trigger RDB persistence:

1. Executing the save command: Executing the save command for persistence will block Redis, and Redis cannot provide external services during the backup.Generally not recommended for use. The usage scenario is when the Redis server needs to be shut down for maintenance.
2. Execute the bgsave command: the bgsave command will not block the Redis main process, and Redis can still provide services to the outside world during the persistence period
3. Triggered by save rules configured in the configuration file:
   

save 900 1: If a key changes within 900s, an RDB snapshot will be triggered

save 300 10: If 10 keys change within 300s, an RDB snapshot is triggered.

save 60 10000: If 10000 keys change (new, modified, deleted) within 60s, the RDB snapshot will be triggered

save "": This configuration indicates turning off RDB persistence

2.2.1.1 RDB persistence principle

When Redis performs RDB, it will fork a child process for data persistence, which will not prevent Redis from continuing to provide external services and improve efficiency

2.2.1.2 RDB advantages and disadvantages

advantage:

• RDB is a very compact file that saves a data set at a certain point in time and is very suitable for backup of data sets. For example: you can save the data of the past 24 hours every hour. At the same time, data for the past 30 days is saved every day. This way, even if something goes wrong you can restore to a different version of the data set if needed.
• RDB When saving an RDB file, the only thing the parent process needs to do is to fork out a child process. All subsequent work is done by the child process. The parent process does not need to perform other IO operations, so the RDB persistence method can maximize the performance of Redis.
• Compared with AOF, the RDB method will be faster when restoring large data sets

shortcoming:

• If the backup interval is long, RDB will lose more data. For example, if a backup is performed at 8:00 and the server goes down at 8:30, the data within half an hour will be lost.

2.2.2 AOF

AOF persistence records each Redis write operation through logs. When the server restarts, these commands will be re-executed to restore the original data. The AOF command uses the Redis protocol to append and save each write operation to the end of the file.

Redis can also rewrite the AOF file in the background, so that the size of the AOF file will not be too large.

AOF persistence configuration:

# 是否开启 aof no：关闭；yes: 开启
appendonly no

# aof 文件名
appendfilename "appendonly.aof"

# aof 同步策略
# appendfsync always  # 每个命令都写入磁盘，性能较差
appendfsync everysec  # 每秒写一次磁盘，Redis 默认配置
# appendfsync no      # 由操作系统执行，默认Linux配置最多丢失30秒

# aof 重写期间是否同步
no-appendfsync-on-rewrite no

# 重写触发策略
auto-aof-rewrite-percentage 100 # 触发重写百分比 （指定百分比为0，将禁用aof自动重写功能）
auto-aof-rewrite-min-size 64mb # 触发自动重写的最低文件体积（小于64mb不自动重写）

# 加载aof时如果有错如何处理
# 如果该配置启用，在加载时发现aof尾部不正确是，会向客户端写入一个log，但是会继续执行，如果设置为 no ，发现错误就会停止，必须修复后才能重新加载。
aof-load-truncated yes

# aof 中是否使用 rdb
# 开启该选项，触发AOF重写将不再是根据当前内容生成写命令。而是先生成RDB文件写到开头，再将RDB生成期间的发生的增量写命令附加到文件末尾。
aof-use-rdb-preamble yes

2.2.2.1 AOF file writing

The aof file is a way to append commands. First write the command into the buffer, and then write it to the disk when the time is up:

appendfsync always    # 每个命令都写入磁盘，性能较差
appendfsync everysec  # 每秒写一次磁盘，Redis 默认配置
appendfsync no        # 由操作系统执行，默认Linux配置最多丢失30秒

The above configuration is when to write to disk

2.2.2.2 AOF rewriting

Although the aof file loses less data, as time goes by, the aof file becomes larger and larger, takes up more and more disk space, and takes a long time to recover. Therefore, redis will rewrite the aof file to reduce the size of the aof file:

-- 重写前的 aof
set k1 20
set k2 40
set k1 35
set k3 34
set k2 19

-- 这里 k1 最终的值为 35，k2 最终值为 19，所以不需要写入两个命令
-- 重写后
set k1 35
set k3 34
set k2 19

2.2.2.3 Hybrid persistence

Starting from Redis version 4.0, a hybrid persistence mechanism has been introduced, pure AOF mode, RDB+AOF mode. This strategy is controlled by the configuration parameter aof-use-rdb-preamble (use RDB as the first half of the AOF file) and is turned off by default ( no), set to yes to enable

• no: Write commands in AOF format, no different from versions before 4.0;
• yes: First write the data status in RDB format, and then write the contents of the AOF buffer during rewriting in AOF format. The first half of the file is in RDB format, and the second half is in AOF format.

The advantages of hybrid persistence are as follows:

• Greatly reduces the size of aof files
• Speeds up the recovery of aof files, preceded by rdb, the recovery speed is fast

2.2.2.4 AOF data recovery

There are two types of AOF data recovery:

• Pure AOF: During recovery, take out the commands in AOF and execute recovery one by one.
• RDB + AOF: Execute the RDB loading process first. After the execution is completed, take out the remaining commands and start executing them one by one.

2.2.2.5 AOF advantages and disadvantages

advantage

• AOF has better real-time performance and less data loss
• AOF already supports hybrid persistence, the file size can be effectively controlled, and the efficiency of data loading is improved.

shortcoming

• For the same data collection, AOF files are usually larger than RDB files
• Under certain fsync strategies, AOF will be slightly slower than RDB
• AOF recovery speed is slower than RDB

3. Redis cluster

In the production environment, we usually use cluster mode when using Redis , because the stand-alone version of Redis has low stability and reliability and limited storage space.

Redis supports three cluster modes:

• master-slave replication
• sentinel mode
• Cluster mode

3.1 Master-slave replication

In the master-slave replication mode, there is one master and multiple slaves, so as to realize the separation of reading and writing. The host is responsible for writing requests, and the slave is responsible for reading requests, reducing the pressure on the host

3.1.1 Master-slave replication principle

The master-slave replication process is as follows:

1. After the slave database is successfully started, connect to the master database and send the SYNC command;
2. After the primary database receives the SYNC command, it starts to execute the BGSAVE command to generate the RDB file and uses the buffer to record all write commands executed thereafter;
3. After the master database BGSAVE is executed, snapshot files are sent to all slave databases, and the executed write commands continue to be recorded during the sending period;
4. After receiving the snapshot file from the database, discard all old data and load the received snapshot;
5. After the master database snapshot is sent, the write command in the buffer begins to be sent to the slave database;
6. The slave database completes loading the snapshot, starts receiving command requests, and executes write commands from the master database buffer; (slave database initialization completed)
7. Every time the master database executes a write command, it will send the same write command to the slave database, and the slave database will receive and execute the received write command (the operation after the slave database initialization is completed)
8. After disconnection and reconnection, versions after 2.8 will pass the commands during the disconnection to the re-database for incremental replication.
9. When the master-slave is just connected, full synchronization is performed; after full synchronization, incremental synchronization is performed. Of course, if necessary, the slave can initiate full synchronization at any time. Redis's strategy is, no matter what, it will first try to perform incremental synchronization. If it is unsuccessful, the slave machine is required to perform full synchronization.

3.1.2 Advantages and disadvantages of master-slave replication

advantage:

• Support master-slave replication, the master will automatically synchronize data to the slave, and can perform read-write separation
• Slave can also accept connection and synchronization requests from other Slaves, which can effectively offload the synchronization pressure of Master
• Master Server provides services for Slaves in a non-blocking manner. So during Master-Slave synchronization, the client can still submit queries or modification requests

shortcoming

• The master never has fault tolerance and recovery capabilities. Once the host hangs up, the entire cluster will be in a readable state and unable to process write requests, and data will be lost.
• After the host is down, it cannot be automatically restored, and can only be restored manually
• The cluster storage capacity is limited. The online capacity is the size of the memory of the main database. More content cannot be stored.

3.2 Sentinel cluster

The role of the sentry is to monitor. Once a problem occurs in the Redis cluster, Sentinel will immediately take corresponding actions to deal with the abnormal situation.

Sentinel mode is built on the master-slave replication mode, because in the master-slave replication mode, the master server goes down, which will cause the entire cluster to be unavailable and require manual intervention. Therefore, the sentinel mode introduces sentinels in the master-slave replication mode to monitor the entire cluster. The architecture diagram of the sentinel mode is as follows:

Sentinel functions:

• Monitoring: Sentry will constantly check whether the master node and slave nodes are functioning properly.

• Automatic failover: When the master node fails to work properly, Sentinel will start an automatic failover operation. It will upgrade one of the slave nodes of the failed master node to the new master node, and let other slave nodes change to replicate the new master node. the master node.

• Configuration provider (Configuration provider): When the client initializes, it obtains the address of the master node of the current Redis service by connecting to the sentinel.

• Notification: Sentry can send the result of failover to the client.

3.2.1 Offline judgment

Redis offline is divided into subjective offline and objective offline

• Subjective offline: the main database of a single Sentinel mission is unavailable
• Objective offline: More than half of the sentinels in the entire sentinel cluster believe that the main database is in a dispensable state.

When any server in the Sentinel cluster determines that the main database is unavailable, it will send a command to other servers in the Sentinel cluster to confirm. After receiving the command, other servers will confirm the status of the main database. If it is unavailable, it will return YES, if it is available, it will return NO, when half of the servers return YES, it means that the main database is really unavailable, and a new election is required:

3.2.2 Main database election

When the Sentinel cluster determines that the main database is offline, a new main database needs to be re-elected to provide external services. So which sentinel should complete the new library election and switching?

Note: Every sentinel cannot be elected here. It may happen that the new main database elected by each sentinel is different, which makes it impossible to determine, so you need to send a representative ------ Sentinel Representative Selection

3.2.3 Sentinel representative selection

The election mechanism of Sentinel is actually very simple. It is a Raft election algorithm: When the number of votes in the election is greater than or equal to num(sentinels)/2+1, it will become the leader. If it does not exceed, the election will continue.

Any Sentinel who wants to become a Leader must meet two conditions:

1. First, get more than half of the votes in favor;
2. Second, the number of votes received needs to be greater than or equal to the quorum value in the sentinel configuration file.

Taking 3 sentinels as an example, assuming that the quorum at this time is set to 2, then any sentry who wants to become a leader only needs to get 2 votes in favor.

3.2.4 New library selection

The sentinel representatives have been elected above. At this time, the representatives need to complete the selection of the new main database. The selection of the new database needs to meet the following criteria:

1. The new library needs to be in a healthy state, that is, to maintain a normal network connection with Sentinel.
2. Select the salve-priority slave node with the highest priority (redis.conf)
3. Select the largest replication offset and only replicate the most complete slave node.

3.2.5 Failover

Failover requires switching between the old and new main databases.

The failover process is as follows:

3.2.6 Advantages and Disadvantages of Sentry Mode

advantage

• Implemented cluster monitoring, failover, and high availability
• Has all the advantages of master-slave replication model

shortcoming

• The cluster storage capacity is limited. The online capacity is the size of the memory of the main database. More content cannot be stored.

3.3 Cluster

The sentinel mode of Redis achieves high availability, but the same data is stored on each Redis server, which wastes memory and makes it difficult to expand capacity. Therefore, the Cluster cluster mode was added to redis3.0 to realize distributed storage of Redis.In other words, different content is stored on each Redis node.

3.3.1 Data sharding of Redis cluster

Redis cluster does not use consistent hashing, but introduces the concept of hash slots.

The Redis cluster has 16384 hash slots. Each key is verified by CRC16 and modulo 16384 is used to determine which slot to place. Each node in the cluster is responsible for a portion of hash slots.

For example: If the current cluster has 3 nodes, then:

• Node A contains hash slots 0 to 5500.
• Node B contains hash slots 5501 to 11000.
• Node C contains hash slots 11001 to 16384.

This structure makes it easy to add or remove nodes. For example, if I want to add a new node D, I need to get some slots from nodes A, B, and C to D. If I want to remove node A, I need to move the slots in A to nodes B and C, and then remove the A node without any slots from the cluster.

Since moving hash slots from one node to another does not stop the service, adding, deleting, or changing the number of hash slots on a node will not cause the cluster to become unavailable.