Redis's high reliability: 1: Guarantee data loss as little as possible; (AOF and RDB guarantee data loss as little as possible) 2: Service interruption as little as possible. (The least interruption of service needs to be achieved by adding redundancy, saving a copy of data in multiple instances. If one instance fails, it will take time to recover, and the rest of the instances can also be used without waiting.)

What is high availability?

High availability: Try not to lose data and provide all services as much as possible.

1. Master-slave architecture data synchronization principle

Redis is down, you can use AOF or RDB to restore the data, by playing back logs and

Redis is down, how to achieve high availability?

Redis provides a master-slave mode, the master node, and the slave node replicates the data backup to other servers redundantly

1. Overview of master-slave replication

effect:

Fault repair: When the master node is down, other nodes can still provide services;
Load balancing: the master node provides write services, and the slave nodes provide read services to share the pressure;
Highly available implementation: the basis for sentinel and cluster implementations

How does the master and slave maintain consistency?

Redis provides the form of a master-slave library to maintain the consistency of data copies, and the read-write separation method is adopted between the master-slave library

The method of reading and writing separation is adopted :

Read operation : both master and slave libraries can be executed;

Write operation : the main library is executed first, and then synchronized to the font library.

The purpose of reading and writing separation?

In order to avoid that the master and slave servers are executing the write operation, each modification is sent to the master and slave instances, which will cause data inconsistency in the instance replicas.

You can imagine that if in the above figure, both the main library and the slave library can receive the write operation of the client, then a direct problem is: if the client modifies the same data (such as k1) three times , each modification request is sent to a different instance, in

If it is executed on different instances, then the copies of this data on these three instances are inconsistent (v1, v2, and v3 respectively). When reading this data, it is possible to read the old value.

If you want to make their data the same, you must lock them, and the instances need to negotiate whether to modify them, and locking will generate additional overhead.

Only the master library can write, and it is synchronized to the slave library without negotiation, and the data remains consistent.

2. Build a master-slave service architecture

3. Master-slave replication principle

The master library first executes the write operation, and then synchronizes to the slave library to ensure data consistency.

3.1 Three cases of synchronization

The first master-slave library full copy
Synchronization during normal operation of master and slave
The network disconnection between the master and slave libraries is synchronized

3.1.1 The first master-slave library full copy

The first copy process is mainly divided into three stages: the connection establishment stage (that is, the preparation stage), the master library synchronizes data to the slave library stage, and sends the new write command in the synchronization stage to the slave library stage

For example: now there are instance 1 (ip: 172.16.19.3) and instance 2 (ip: 172.16.19.5), we execute the following command on instance 2 to make instance 2 a slave library of instance 1 and copy data:

replicaof 172.16.19.3 6379

establish connection

The first stage

The main function of this stage is to establish a connection between the master and slave nodes to prepare for full data synchronization. The slave library will establish a connection with the master library, and the slave library (execute replicaof) sends the psync command to the master library and tells the master library that synchronization is about to take place. After the master library confirms the reply, the synchronization between the master and slave libraries begins . ( The first stage is the process of establishing a connection between the master and slave libraries and negotiating synchronization, mainly to prepare for full replication. In this step, the slave library and the master library establish a connection, and tell the master library that synchronization is about to take place, and the master library confirms the reply After that, the master-slave library can start to synchronize .)

How does the slave library know the information of the main library and establish a connection?

After configuring the IP and port of the master node in the replicaof configuration item in the configuration file of the slave node, the slave node knows that it wants to connect to the master node.

Two fields are maintained inside the slave node, masterhost and masterport, which are used to store the IP and port information of the master node.

The slave library (executes replicaof) sends the psync command to the master library, indicating that data synchronization is to be performed, and the master library starts replication according to the parameters after receiving the command. The command contains two parameters , the runID of the master library and the copy progress offset .

runID : Each Redis instance startup will automatically generate a unique ID. For the first master-slave replication, the master database runID is not yet known, and the parameter is set to "?".
offset : The first copy is set to -1, which means the first copy, and the offset of the copy progress is recorded.

After the main library receives the psync command, it will use the FULLRESYNC response command with two parameters: the main library runID and the current replication progress offset of the main library, and return it to the slave library . These two parameters are logged when the response is received from the library.

The FULLRESYNC response indicates the full copy used for the first copy , that is, the master library will copy all the current data to the slave library. resync

The master library synchronizes data to the slave library

Why use RDB not use AOF file?

AOP files are larger than RDB files, and network transmission is time-consuming.

RDB performs faster than AOF file when initializing data from library

second stage

The master library synchronizes all data to the slave library. After receiving the data from the library, complete the data loading locally. This stage relies on the RDB generated by the memory snapshot.

The master executes the bgsave command to generate an RDB file, and sends the file to the slave library. At the same time, the master library creates a replication buffer buffer for each slave to record all write commands received from the generation of the RDB file.

After receiving the RDB file from the library, save it to the disk, clear the data in the current database, and then load the RDB file data into the memory. (Save the disk first, clear the memory data, and finally load rdb into the memory)

Specifically, the master library executes the bgsave command to generate an RDB file, and then sends the file to the slave library. After receiving the RDB file from the library, the current database will be cleared first, and then the RDB file will be loaded. This is because the slave library may have saved other data before starting to synchronize with the master library through the replicaof command. In order to avoid the impact of previous data, the slave library needs to clear the current database first.

During the process of synchronizing data from the main library to the slave library, the main library will not be blocked and can still receive requests normally. Otherwise, the Redis service is interrupted. However, the write operations in these requests are not recorded in the RDB file just generated. In order to ensure the data consistency of the master-slave library, the master library will use a special replication buffer in the memory to record all the write operations received after the RDB file is generated.

Send a new write command to the slave library

The third phase

After the slave node loads the RDB, the master node sends the data in the replication buffer buffer to the slave node, the slave receives and executes it, and the slave node synchronizes to the same state as the master node.

Finally, in the third stage, the master library will send the newly received write commands during the execution of the second stage to the slave library. The specific operation is that when the main library finishes sending the RDB file, it will send the modification operations in the replication buffer to the slave library, and the slave library will re-execute these operations. In this way, the master-slave library is synchronized.

Receive the request normally

The write operation after the RDB file is generated is not recorded in the RDB file just now. In order to ensure the consistency of the master-slave database data, the master library will use a replication buffer in memory to record all the write operations after the RDB file is generated.

Do you want to clear the current database after receiving the RDB file from the library?

Because the slave library may save other data before starting to synchronize with the master library through the replicaof command, to prevent the impact between the master and slave data.

Replication buffer explained?

A buffer created on the master side, the stored data is all master data write operations in the following three periods.

1) The master executes bgsave to generate the write operation during the RDB generation;

2) The master sends rdb to the write operation during the slave network transmission;

3) The write operation during which the slave load rdb file restores the data to the memory.

After each client (client communication, slave library communication) connects to Redis, Redis will allocate a dedicated client buffer, and all data interactions are carried out through this buffer. (The Master first writes the data into this buffer, and then sends it out through the network, thus completing the data interaction). The buffer is specially used to propagate the write command to the slave library to ensure the consistency of the master-slave data. We usually call it the replication buffer.

Problems caused by too small replication buffer:

The replication buffer is set by client-output-buffer-limit slave. When this value is too small, the master-slave replication connection will be disconnected . (The replication buffer is set by client-output-buffer-limit slave. When this value is too small, the master-slave replication connection will be disconnected .)

Problems caused by the disconnection of the master-slave replication connection:

config set client-output-buffer-limit "slave 536870912 536870912 0"

1) When the master-slave replication connection is disconnected, the master will release the data related to the connection. The data in the replication buffer is also lost, and the replication process is restarted between the master and the slave.

2) There is a more serious problem. The master-slave replication connection is disconnected, resulting in an infinite loop of re-executing bgsave and rdb retransmission operations on the master-slave.

When the data volume of the master node is large, or the network delay between the master and slave nodes is large, the size of the buffer may exceed the limit, and the master node will disconnect from the slave node at this time;

The problem caused by the large amount of master node data: (the master-slave connection is disconnected. Infinite loop)

This situation may cause full copy -> replication buffer overflow leads to connection interruption -> reconnection -> full copy -> replication buffer buffer overflow leads to connection interruption... cycle.

Why doesn't master-slave library replication use AOF? Compared with RDB, less data is lost

RDB files are binary files, and the IO efficiency of network transmission RDB and writing to disk is higher than that of AOF.
When recovering data from the database, the recovery efficiency of RDB is also higher than that of AOF.

3.1.2 The second type: Incremental replication (synchronization of network disconnection)

What should I do if the network between the master and slave databases is broken?

Before 2.8, disconnecting and re-doing a full copy will cost a lot of money. After 2.8, disconnect and reconnect, and the master-slave will continue to synchronize incrementally.

Full copy: Synchronize all data

Incremental replication: It is used for replication after network interruption and other situations. Only the write commands executed by the master node during the interruption are sent to the slave nodes, which is more efficient than full replication .

When the master-slave library is disconnected, the master library will write the write operation commands received during the disconnection period into the replication buffer, and will also write these operation commands

Also write to the buffer repl_backlog_buffer.

buffer used incrementally

repl_backlog_buffer

Main library: record the location you wrote

From the library: record the position you read

The secret of disconnected and reconnected incremental replication is the repl_backlog_buffer buffer. Whenever the master will record the write instruction operation in repl_backlog_buffer, because of limited memory, repl_backlog_buffer is a fixed-length circular array. If the array is full, it will will overwrite the previous content from scratch .

At the beginning, the write and read positions of the master library and the slave library are together, which is their starting position. As the main library continues to receive new write operations, its write position in the buffer will gradually deviate from the initial position. We usually use the offset to measure the size of this offset distance. For the main library, the corresponding offset The displacement is master_repl_offset. The more new write operations the main library receives, the larger this value will be.

Similarly, after the slave library finishes copying the write operation command, its read position in the buffer also begins to gradually shift from the initial position just now. At this time, the offset slave_repl_offset that has been copied from the library is also increasing. Under normal circumstances, these two offsets are basically equal.

The master uses master_repl_offset to record the position offset it has written, and the slave uses slave_repl_offset to record the offset it has read.

The master receives the write operation, and the offset is incremented. After the slave library continues to execute synchronous write instructions, slave_repl_offset is also increasing.

Under normal circumstances, these two offsets are basically equal. During the network disconnection stage, the main library may receive new write operation commands, so master_repl_offset will be greater than slave_repl_offset.

When the master-slave is disconnected and reconnected, the slave will first send the psync command to the master, and at the same time send its own runID, slave_repl_offset to the master. The main library will judge the gap between its own master_repl_offset and slave_repl_offset.

At this point, the master only needs to synchronize the commands between master_repl_offset and slave_repl_offset to the slave library.

Just like the middle part of the schematic diagram, there are two operations of put de and put df between the master library and the slave library. During incremental replication, the master library only needs to synchronize them to the slave library.

However, there is one point I want to emphasize, because repl_backlog_buffer is a ring buffer, so after the buffer is full, the main library will continue to write, and at this time, the previously written operation will be overwritten. If the reading speed of the slave library is relatively slow, it may cause the unread operations of the slave library to be overwritten by the new write operation of the master library, which will lead to data inconsistency between the master and slave libraries.

Repl_backlog_buffer expansion:

If the repl_backlog_buffer is too small, what should I do if it is overwritten by the master's new write operation before it is read from the library?

We have to find a way to avoid this situation, and once it is overwritten, a full copy will be performed. We can adjust the repl_backlog_size parameter to control the buffer size. Calculation formula:

repl_backlog_buffer = second * write_size_per_second

second : the average time required to disconnect from the server and reconnect to the master server;
write_size_per_second : The average amount of command data generated by the master per second (sum of write command and data size);

For example, if the master server writes an average of 1 MB per second, and it takes an average of 5 seconds for the slave to reconnect to the master after a disconnection, then the size of the replication backlog buffer cannot be lower than 5 MB.

To be on the safe side, the size of the replication backlog buffer can be set to 2 * second * write_size_per_second, which ensures that most disconnections can be handled with partial resync.

repl_backlog_buffer =2 * second * write_size_per_second

In this way, the risk of data inconsistency between the master and slave databases during incremental replication is reduced. However, if the amount of concurrent requests is very large, and even twice the buffer space cannot store new operation requests, at this time, the master-slave database data may still be inconsistent.

For this situation, on the one hand, you can appropriately increase the repl_backlog_size value according to the memory resources of the server where Redis is located, for example, set it to 4 times the size of the buffer space; on the other hand, you can consider using a slice cluster to share the burden of a single main library Ask for pressure.

How to determine whether to perform a full synchronization or a partial synchronization?

In Redis 2.8 and later, the slave node can send the psync command to request data synchronization. At this time, depending on the current status of the master and slave nodes, the synchronization method may be full copy or partial copy

Full copy or incremental copy judgment:

The key is the execution of psync:

The slave node sends the psync command to the master according to the current state:

- If the slave node has never executed replicaof, the slave node sends psync ? -1 to send a full copy request to the master node;
- If the slave node has executed replicaof before, it will send psync <runID> <offset>, where runID is the runID of the master node saved in the last replication, and offset is the replication offset saved by the slave node when the last replication ended.

The master node decides to perform full or partial replication based on the received psync command and the current server status:

- The runID is the same as the runID sent by the slave node, and the data after slave_repl_offset sent by the slave node exists in the repl_backlog_buffer buffer, then reply CONTINUE, indicating that partial replication will be performed, and the slave node can wait for the master node to send the missing data;
- The runID is different from the runID sent by the slave node, or the data after slave_repl_offset sent by the slave node is no longer in the repl_backlog_buffer buffer of the master node (extruded in the queue), then reply to the slave node FULLRESYNC <runid> <offset>, indicating To perform full replication, runID represents the current runID of the master node, offset represents the current offset of the master node, and the slave node saves these two values for future use.

If a slave library is disconnected from the main library for too long, its data at the slave_repl_offset position of the main library repl_backlog_buffer has been overwritten. At this time, a full copy will be performed between the slave library and the main library.

3.1.3. The third type: command propagation based on long connection

How to synchronize the normal operation process after the full synchronization is completed

Command propagation based on long connections: When the master-slave library completes full replication, a network connection will be maintained between them, and the master library will resynchronize the subsequent command operations received successively to the slave library through this connection

Purpose: The purpose of using long connections is to avoid the overhead caused by frequent connection establishment.

In the command propagation phase, in addition to sending write commands, the master and slave nodes also maintain a heartbeat mechanism: PING and REPLCONF ACK.

The master-slave cascade mode shares the pressure of the master library during full replication

By analyzing the first data synchronization process between the master and slave databases, you can see that in a full copy, for the master database, two time-consuming operations need to be completed: generating RDB files and transferring RDB files.

If there are a large number of slave libraries, and they must be fully replicated with the main library, the main library will be busy with the fork child process to generate RDB files and perform full data synchronization. The fork operation will block the main thread from processing normal requests, which will slow down the main library's response to application requests. In addition, transferring RDB files will also occupy the network bandwidth of the main library, which will also put pressure on the resource usage of the main library. So, is there a good solution to share the pressure on the main library?

In fact, there is, this is the "master-slave-slave" mode.

In the master-slave library model just introduced, all slave libraries are connected to the master library, and all full replication is also performed with the master library. Now, we can use the "master-slave-slave" mode to distribute the pressure of generating RDB and transmitting RDB from the master library to the slave library in a cascading manner.

To put it simply, when we deploy a master-slave cluster, we can manually select a slave library (for example, select a slave library with a higher memory resource configuration) to cascade other slave libraries . Then, we can select some slave libraries (for example, one-third of the slave libraries), and execute the following commands on these slave libraries, so that they can establish a master-slave relationship with the slave library just selected.

The slave library ip port selected by replicaof

In this way, these slave libraries will know that when synchronizing, they no longer need to interact with the main library. They only need to synchronize write operations with the cascaded slave libraries, which can reduce the pressure on the main library, as follows As shown in the picture:

Heartbeat mechanism: (check if the network is disconnected)

Master->Slave: PING

Every once in a while, the master library will send a ping command to the slave library for the purpose of allowing the slave node to judge the connection timeout.

Slave -> Master: REPLCONF ACK

In the command propagation phase , the slave server will send commands to the master server once per second by default:

REPLCONF ACK <replication_offset>

where replication_offset is the current replication offset from the server. Sending the REPLCONF ACK command has three functions for the master and slave servers: replconf ack

Detect the network connection status of the master and slave servers.
Auxiliary implementation of the min-slaves option.
To detect the loss of the command, the slave node sends its own slave_replication_offset, and the master node will compare it with its own master_replication_offset. If the slave node data is missing, the master node will find and push the missing data from the repl_backlog_buffer buffer. Note that offset and repl_backlog_buffer buffers can be used not only for partial replication, but also for dealing with situations such as command loss; the difference is that the former is performed after disconnection and reconnection, while the latter is when the master and slave nodes are not disconnected carried out below.

3.1.3 Summary

summary

After learning the basic principle of Redis master-slave library synchronization, in summary, there are three modes: full copy, command propagation based on long connection, and incremental copy.

Although full replication is time-consuming, full replication is unavoidable for the slave library if it is the first synchronization. Therefore, I will give you a small suggestion: the database of a Redis instance should not be too large, and the size of an instance should be several The GB level is more appropriate, which can reduce the overhead of RDB file generation, transmission and reloading. In addition, in order to avoid multiple simultaneous full replications of the master library and excessive synchronization pressure on the master library, we can also use the "master-slave-slave" cascading mode to relieve the pressure on the master library.

Long connection replication is a regular synchronization stage after the master-slave database is running normally. In this stage, the synchronization between the master and slave libraries is achieved through command propagation. However, if there is a network disconnection during this period, incremental replication will come in handy. In particular, I recommend that you pay attention to the repl_backlog_size configuration parameter. If it is configured too small, during the incremental replication phase, the replication progress of the slave library may not catch up with the master library, which will cause the slave library to perform full replication again. Therefore, by increasing this parameter, the risk of full copying from the library when the network is disconnected can be reduced.

However, although the master-slave library mode uses read-write separation to avoid data inconsistency caused by writing multiple instances at the same time, it still faces the potential risk of master library failure.

Basic summary

Each slave library will record its own slave_repl_offset, and the replication progress of each slave library is not necessarily the same. (The slave library will record the received offset, and the copy progress of each slave library may be inconsistent)

When reconnecting with the main library for recovery, the slave library will send the slave_repl_offset recorded by itself to the main library through the psync command, and the main library will determine whether the slave library can perform incremental replication or full copy according to the respective replication progress of the slave library copy. (According to the runid and offset offerset data sent by psync to determine whether to perform full copy)

replication buffer 和 repl_backlog_buffer

The replication buffer corresponds to each slave and is set by config set client-output-buffer-limit slave.
repl_backlog_buffer is a ring buffer, there will only be one in the entire master process, and all slaves are common. The size of repl_backlog is set by the repl-backlog-size parameter, the default size is 1M, and its size can be based on the command generated per second, (master executes rdb bgsave) + (master sends rdb to slave) + (slave load rdb file) time and to estimate the size of the backlog buffer, the repl-backlog-size value is not less than the product of these two.

In general, the replication buffer is the buffer used on the master library for clients connected to the slave library when the master-slave library is performing full replication, while the repl_backlog_buffer is used to support incremental replication from the library, and is used to continuously save writes on the master library. A dedicated buffer for the operation.

When the Redis master-slave library is replicating, when the master library wants to send the write operation command during the full copy period to the slave library, the master library will first create a client (replication buffer) to connect to the slave library, and then pass this client , and send the write operation command to the slave library. In the memory, the client on the main library will correspond to a buffer, which is called the replication buffer. Redis controls the size of this buffer through the client_buffer configuration item. The master library will create a client for each slave library, so the replication buffer is not shared, but each slave library has a corresponding client.

repl_backlog_buffer is a dedicated buffer. After the Redis server starts, it starts to receive write operation commands all the time, which is shared by all slave libraries. The master library and the slave library will each record their own replication progress, so when different slave libraries are recovering, they will send their own replication progress (slave_repl_offset) to the master library, and the master library can synchronize with it independently.

5. Master-slave application problem

Under master-slave replication, will the slave node delete expired data?

For the data consistency of master and slave nodes, slave nodes will not actively delete data

5.1 Redis has two deletion strategies:

Lazy deletion: When the client queries the corresponding data, Redis judges whether the data is expired, and deletes it if it expires.
Periodic deletion: Redis deletes expired data through scheduled tasks.

Will the client read expired data by reading data from the node?

Starting from Redis 3.2, when reading data from a node, first determine whether the data has expired. If it expires, it will not return to the client and delete the data.

5.2 Stand-alone memory size limit

If the Redis stand-alone memory reaches 10GB, the synchronization time of a slave node is at the level of several minutes; if there are more slave nodes, the recovery speed will be slower. If the read load of the system is high, and the slave nodes cannot provide services during this period, it will put a lot of pressure on the system.

If the amount of data is too large, it takes too long for the master node to fork + save the RDB file during the full copy phase, and the slave node cannot receive data for a long time to trigger a timeout. The data synchronization of the master and slave nodes may also fall into full copy -> timeout leads to replication interruption- >Reconnect->Full copy->Timeout causes the copy to be interrupted ... cycle.

In addition, in addition to the absolute amount of the master node's single-machine memory, it should not be too large, and its proportion of the host's memory should not be too large: it is best to use only 50% - 65% of the memory, leaving 30% - 45% of the memory for bgsave commands and create copy buffers etc.

6. Summary

The role of master-slave replication: AOF and RDB binary files ensure fast recovery of data during downtime, and prevent data loss as much as possible. However, services cannot be provided after a downtime, so a master-slave architecture and read-write separation have evolved.
The principle of master-slave replication: connection establishment phase, data synchronization phase, and command propagation phase; the data synchronization phase is divided into full replication and partial replication; in the command propagation phase, there are PING and REPLCONF ACK commands between the master and slave nodes to perform mutual heartbeat detection.
Although master-slave replication solves or alleviates problems such as data redundancy, fault recovery, and read load balancing, its defects are still obvious: fault recovery cannot be automated; write operations cannot be load balanced; storage capacity is limited by a single machine; the solution to these problems , need the help of sentry and cluster, I will introduce in the following article, welcome to pay attention.

2. Problems encountered in Redis master-slave synchronization and failover

The master-slave synchronization mechanism of Redis not only allows the slave library to serve more read requests and share the pressure of the main library, but also performs master-slave library switching to provide highly reliable services when the main library fails.

When the master-slave mechanism is actually used, three problems will be encountered: the master-slave data is inconsistent, expired data is read, and the configuration item is set unreasonably, which leads to the service hanging.

1. Inconsistent master-slave data

Master-slave data inconsistency means that the value read by the client from the slave library is inconsistent with the latest value in the master library.

1.1 Example

Assume that the user age value previously saved by the master-slave library is 19, but the master library has received a modification command and has updated this data to 20, but the value in the slave library is still 19. Then, if the client reads the user age value from the repository, it will read the old value.

1.2 Main reasons:

Command replication between the master and slave libraries is performed asynchronously .

Specifically, in the master-slave library command propagation phase, after the master library receives a new write command, it will send it to the slave library. However, the main library does not wait until the slave library actually executes the command, and then returns the result to the client, but the main library itself returns the result to the client after executing the command locally. If the slave library has not executed the command synchronized from the master library, the data between the master and slave libraries will be inconsistent.

1.3 The reason why the slave library will lag behind the execution of synchronization commands

Reason 1: Network transmission delay

The network between the master and slave libraries may have transmission delays, so the slave library cannot receive the commands sent by the master library in time, and the execution time of the synchronization command on the slave library will be delayed

Reason 2: Blocked

Even if the slave library receives the command from the master library in time, it may be blocked because it is processing other high-complexity commands (such as set operation commands). The slave library needs to finish processing the current command before executing the command operation sent by the master library, which will cause the master-slave data inconsistency. During the period when the main library command is delayed, the main library itself may perform new write operations. In this way, the degree of data inconsistency between the master-slave database will be further exacerbated.

1.4 Solutions

Method 1: Make the configuration network good

In terms of hardware environment configuration, we should try our best to ensure that the network connection between the master and slave libraries is in good condition . For example, we need to avoid deploying master-slave libraries in different computer rooms, or avoid deploying network-intensive applications (such as data analysis applications) and Redis master-slave libraries together.

Method 2: External Program Helps Intervene

Develop an external program to monitor the progress of replication between master and slave libraries

Because the INFO replication command of Redis can view the progress information of the master library receiving the write command (master_repl_offset) and the progress information of the slave library copying the write command (slave_repl_offset), you can develop a monitoring program, first use the INFO replication command to check the master and slave library. Progress, subtract slave_repl_offset from master_repl_offset to get the difference in replication progress between the slave library and the master library.

If the progress difference of a slave library is greater than the preset threshold, the client will no longer connect to the slave library to read data, reducing the situation of reading inconsistent data. In order to avoid the situation that the client and all slave libraries cannot connect, it is necessary to set a larger threshold for the replication progress difference.

This process can be run periodically to monitor inconsistencies between master and slave libraries

The monitoring program can always monitor the copy progress of the slave library, and when the copy progress of the slave library catches up with the main library, the client is allowed to connect to these slave libraries again.

2. Read expired data

When using a Redis master-slave cluster, sometimes outdated data is read. For example, the expiration time of data X is 202010240900, but the client can still read data X from the library at 202010240910

2.1 The reason why the client can read expired data

This is caused by Redis' expired data deletion strategy. Redis uses two strategies to delete expired data at the same time, namely the lazy deletion strategy and the regular deletion strategy.

Lazy deletion strategy: not deleted immediately after expiration, deleted when there is a request to read again

When an expired key is accessed, Redis checks whether the key is expired. If expired, it will be deleted immediately. If it is not expired, Redis will continue to process it like a normal key. This approach is called lazy deletion, because Redis only checks for expiration when a key needs to be accessed, and deletes it when it finds it expired.

The advantage of this strategy is to minimize the use of CPU resources by delete operations, and no longer waste time checking and deleting unused data. However, this strategy will cause a large amount of expired data to remain in memory, occupying more memory resources. Redis uses this strategy while also using the second strategy: regular deletion strategy.

Periodic deletion strategy: delete all expired key values for a period of time

Redis also uses a periodic deletion strategy to delete expired keys. Under this strategy, Redis will scan the database every once in a while, find all expired keys and delete them. This method is called periodic deletion, because Redis will perform this operation periodically

illustrate

The lazy deletion strategy allows Redis to use memory more efficiently because it only checks for expiration when a key needs to be accessed. However, it can cause expired keys to stay in memory for a long time until the next time they are accessed. The periodic deletion strategy can ensure that expired keys are deleted in time, but it may have a certain impact on performance, because it needs to traverse the entire database periodically. In order to balance memory usage and performance, Redis usually uses both strategies at the same time.

The regular deletion strategy can release some memory. In order to avoid the impact of too many deletion operations on performance, Redis randomly checks the amount of data each time is not large. If there is a lot of expired data and it has not been accessed, the data will remain in the Redis instance. The reason why business applications read expired data is that these retained data are an important factor.

After the lazy deletion strategy is implemented, the data will not be actually deleted until it is accessed again. If the client reads the retained expired data from the main library, the main library will trigger the delete operation, and at this time, the client will not read the expired data. However, the slave library itself will not perform the deletion operation. If the client accesses the retained expired data in the slave library, the slave library will not trigger data deletion

Will the slave library return expired data to the client?

Before Redis version 3.2, the slave library would return expired data. After Redis3.2, the slave library will not return expired data, but will return a null value, but it is still possible to read it.

If you are using a version earlier than Redis 3.2, then when the slave library is serving a read request, it will not judge whether the data is expired, but will return the expired data.
After version 3.2, Redis has made improvements. If the read data has expired, although the slave library will not delete it, it will return a null value, which prevents the client from reading expired data. Therefore, when applying master-slave clusters, try to use Redis 3.2 and above.

If you use a version after Redis 3.2, you will still read expired data. This is related to the command Redis uses to set the expiration time. The expiration time set by some commands for the data may be delayed on the slave library, resulting in data that should expire. The data is read from the library again

Description of command to set data expiration time

EXPIRE and PEXPIRE: They set the survival time for the data from the time the command is executed;
EXPIREAT and PEXPIREAT: They will directly set the expiration time of the data to a specific point in time.

example

The first example is to use the EXPIRE command. When the following command is executed, the expiration time of the testkey is set to 60s later.

expire testkey 60

The second example is to use the EXPIREAT command. For example, to let the testkey expire at 9 am on October 24, 2020, the 1603501200 in the command is the timestamp on October 24 at 9 am in seconds.

expireat testkey 1603501200

How the command results in reading outdated data

The master-slave library is fully synchronized. After receiving the EXPIRE command, the master library executes it directly. After the full synchronization is completed, it is sent to the slave library. When the slave library executes, the survival time will be added to the current time, and the expiration time will be significantly delayed.

example

Suppose the current time is 9 am on October 24, 2020, the master-slave library is synchronizing, and the master library has received a command: EXPIRE testkey 60, which means that the expiration time of the testkey is 9:1 am on the 24th, and the master library This command was executed directly.

However, the full synchronization of the master-slave library took 2 minutes to complete. When the slave library started to execute this command, the time was already 9:2. The EXPIRE command is to set the expiration time of the testkey to 60s after the current time, which is 9:3. If the client reads testkey from the library at 9:02:30, it can still read the value of testkey. However, the testkey has actually expired

In order to avoid this situation, use the EXPIREAT/PEXPIREAT command in business applications to set the expiration time of the data to a specific time point to avoid reading expired data

3. Partial summary

Master and slave data are inconsistent. Redis uses asynchronous replication, so it cannot achieve strong consistency guarantee (master-slave data is consistent at all times), and data inconsistency is unavoidable. What needs to be done: ensure a good network environment, use the program to monitor the copy progress of the slave library, once the copy progress of the slave library exceeds the threshold, do not allow the client to connect to the slave library

Reading expired data can be avoided in advance. One method is to use Redis 3.2 and above; you can use the EXPIREAT/PEXPIREAT command to set the expiration time to avoid the data expiration time lag on the slave library. It should be noted that because EXPIREAT/PEXPIREAT sets the time point, the clocks on the master and slave nodes must be consistent. The specific method is to synchronize the clocks of the master and slave nodes with the same NTP server (time server)

4. The service hangs up caused by unreasonable configuration items

Two configuration items are involved: protected-mode and cluster-node-timeout.

1.protected-mode configuration item

The purpose of this configuration item is to limit whether the sentinel instance can be accessed by other servers. When the configuration is set to yes, the sentinel instance can only be accessed by the local server (the deployment of this instance is considered local). When set to no, other servers can also access the sentinel instance.

If protected-mode is set to yes, and the remaining Sentinel instances are deployed on other servers, there will be no communication between these Sentinel instances. The main library fails, Sentinel cannot judge that the main library is offline, and cannot perform master-slave switching, and finally the Redis service is unavailable.

When applying a master-slave cluster, pay attention to setting the protected-mode configuration item to no, and setting the bind configuration item to the IP address of another sentinel instance. Only the sentry with the IP address set in bind can access the current instance, which not only ensures the communication between instances for master-slave switching, but also ensures the security of the sentinel.

example:

If the following configuration items are set, Sentinel instances deployed on the three servers 192.168.10.3/4/5 can communicate with each other and perform master-slave switching.

protected-mode no
bind 192.168.10.3 192.168.10.4 192.168.10.5

2.cluster-node-timeout configuration item

This configuration item sets the timeout period for instances in Redis Cluster to respond to heartbeat messages.

When the "one master and one slave" mode is configured for each instance in the Redis Cluster cluster, if the master instance fails, the slave instance will switch to the master instance. Due to the impact of network delay and switching operation execution, the switching time may be longer. It will cause the heartbeat of the instance to time out (exceeding cluster-node-timeout). After the instance times out, it will be judged as abnormal by Redis Cluster . The condition for the normal operation of Redis Cluster is that more than half of the instances can operate normally.

If more than half of the instances perform master-slave switching, and the master-slave switching time is too long, the heartbeat of more than half of the instances may timeout, which may cause the entire cluster to hang. It is recommended to increase the cluster-node-timeout (for example, 10 to 20 seconds).

5. Summary

Three possible problems may occur when the master-slave library is synchronized: the master-slave data is inconsistent, the expired data is read, and the unreasonable configuration item causes the service to hang up

About the inconsistency of master-slave database data

The slave-serve-stale-data configuration item in Redis sets whether the slave library can handle data read and write commands, set it to no, the slave library can only serve INFO, SLAVEOF commands, and avoid reading inconsistent data from the library .

The difference between the lave-serve-stale-data configuration item and slave-read-only

slave-read-only is to set whether the slave library can process write commands. When slave-read-only is set to yes, the slave library can only process read requests and cannot process write requests

Is it a good way to set slave-read-only to no, so that data can be deleted directly from the library, so as to avoid reading expired data?

Answer: The addition, deletion, and modification operations in master-slave replication need to be performed on the master library. Even if the slave library can be deleted, do not delete from the slave library, otherwise it will cause data inconsistency. The clocks of the master-slave library are not synchronized, resulting in inconsistent deletion time of the master-slave library. expire is delayed

For the key that has already set the expiration time, the main library uses the expire command to reset the expiration time when the key is about to expire. For example, a key is originally set to expire after 10s. The expire command sets the expiration time of the key to 60s later. However, when the expire command is transmitted from the master library to the slave library, due to network delays, the slave library did not receive the expire command in time (for example, the slave library received the expire command after a 3s delay), so the slave library follows the original expiration time The expired key is deleted, which leads to the inconsistency of the master-slave data. (After the master-slave synchronization, the slave library will execute the expire command)

An article to let you know about Redis synchronization--explain all in detail