Detailed explanation of Redis cluster principle

1. Introduction to Redis cluster:

1. Why do we need Redis cluster?

Before talking about the Redis cluster architecture, let's briefly talk about the architecture of Redis single instance, from the initial one master N slave, to read and write separation, and then to the Sentinel sentinel mechanism. The single instance of Redis cache is sufficient for most usage scenarios. , Can also achieve master-slave failure migration.

However, in some scenarios, there are several problems with single-instance storage Redis cache:

(1) Write concurrency:

Redis single-instance read-write separation can solve the load balancing of read operations, but for write operations, all still fall on the master node. In the scenario of massive data with high concurrency, one node is prone to bottlenecks in writing data, causing the pressure on the master node to increase. .

(2) Storage pressure of massive data:

Single-instance Redis essentially only has one Master as storage. If faced with the storage of massive amounts of data, a Redis server cannot cope with it, and too large a data volume means high persistence costs, which may block the server in severe cases. This results in a decrease in the success rate of service requests and reduces service stability.

In response to the above problems, the Redis cluster provides a more complete solution to solve the problem that storage capacity is limited by a single machine and write operations cannot be load balanced.

2. What is a Redis cluster?

Redis3.0 has joined the Redis cluster mode, which realizes the distributed storage of data, shards the data, and stores different data on different master nodes, thus solving the problem of massive data storage.

The Redis cluster adopts the idea of decentralization. There is no central node. For the client, the entire cluster can be regarded as a whole and can be connected to any node for operation, just like operating a single Redis instance, without any agent intermediate When the key operated by the client is not assigned to the node, Redis will return the steering command to point to the correct node.

Redis also has a built-in high availability mechanism that supports N master nodes. Each master node can mount multiple slave nodes. When the master node goes down, the cluster will promote one of its slave nodes as the new master node.

As shown in the figure above, the Redis cluster can be seen as a combination of multiple master-slave architectures, and each master-slave architecture can be seen as a node (among which, only the master node has the ability to process requests, and the slave node is mainly used for high-level nodes. Available)

2. Data distribution algorithm of Redis cluster: hash slot algorithm

1. What is the hash slot algorithm?

As mentioned earlier, Redis cluster solves the problem of massive data storage of a single node through distributed storage. For distributed storage, the key point to consider is how to split the data to different Redis servers. Common partitioning algorithms include hash algorithm and consistent hash algorithm. I won't introduce more about these algorithms here.

Ordinary hash algorithm: After calculating the key using the hash algorithm, take the remainder according to the number of nodes, that is, hash(key)%N. The advantage is that it is relatively simple, but when expanding or removing nodes, you need to recalculate the mapping relationship, which will cause the data to re-migrate.

Consistent hash algorithm: Assign a token to each node to form a hash ring; when searching, first calculate the hash value according to the key, and then clockwise find the first token node greater than or equal to the hash value. The advantage is that only two adjacent nodes are affected when adding and deleting nodes. The disadvantage is that adding or subtracting nodes will cause some data to be missed, so it is generally used for caching, and when the number of nodes is large, the expansion is generally doubled The node guarantees data load balancing.

The algorithm adopted by the Redis cluster is the hash slot partition algorithm. There are 16384 hash slots in the Redis cluster (the range of slots is 0-16383, hash slots). Different hash slots are distributed on different Redis nodes for management, which means that each Redis node is only responsible for part of the Hash slot. When operating on the data, the cluster will use the CRC16 algorithm to calculate the key and take the modulus of 16384 (slot = CRC16(key)%16383). The result is the slot where the Key-Value is placed, and this value is passed , To find the Redis node corresponding to the corresponding slot, and then go directly to the corresponding node for access operations.

The advantage of using a hash slot is that you can easily add or remove nodes, and whether you add or delete a node or modify a node, it will not cause the cluster to become unavailable. When you need to add nodes, you only need to move some of the hash slots of other nodes to the new node; when you need to remove the node, you only need to move the hash slots on the removed node to other nodes; The hash slot data partition algorithm has the following characteristics:

Decouple the relationship between data and nodes, simplifying the difficulty of expansion and contraction;
The node itself maintains the mapping relationship of the slots, and does not require the client agent service to maintain the slot partition metadata
Support mapping query between nodes, slots, and keys for data routing, online scaling and other scenarios

Slot migration and assignment command: CLUSTER ADDSLOTS 0 1 2 3 4 ... 5000

By default, the read and write of the redis cluster are performed on the master, and the slave node does not support read and write. It is different from the separation of read and write under Redis master-slave replication, because the core concept of the redis cluster is mainly to use slave Do hot backup of data, and switch between master and backup when master fails to achieve high availability. Redis's separation of reads and writes is to expand slave nodes horizontally to support greater read throughput. Under the redis cluster architecture, the master itself can be scaled arbitrarily. If you want to support greater read or write throughput, you can directly scale the master horizontally.

2. Data structure related to the hash slot in Redis:

(1) ClusterNode data structure: save the current state of the node, such as the creation time of the node, the name of the node, the current configuration epoch of the node, the IP and address of the node, and so on.

(2) clusterState data structure: record the current state of the cluster that the current node thinks.

(3) Slot assignment information of the node:

The slots attribute and numslot attribute of the clusterNode data structure record which slots the node is responsible for processing:

The slots attribute is a bit array. The length of this array is 16384/8=2048 bytes, which contains 16384 binary bits in total. The Master node uses bits to identify whether it owns a slot, and the time complexity is O(1)

(4) Assignment information of all slots in the cluster:

When receiving information from other nodes in the cluster, by storing the node slot assignment information in the local clusterState.slots array, the program must check whether slot i has been assigned, or get the node responsible for processing slot i. Need to access the value of clusterState.slots[i], the time complexity is only O(1)

As shown in the figure above, each subscript in the Slots array saved in ClusterState corresponds to a slot, and each slot information corresponds to a clusterNode, which is a cached node. These nodes correspond to an actual Redis cache service, including IP and Port information. The communication mechanism of Redis Cluster actually ensures that each node has the corresponding relationship between other nodes and slot data. No matter which node in the cluster is accessed by the Redis client, it can be routed to the corresponding node, because each node has a ClusterState, which records the correspondence between all slots and nodes.

3. Request redirection of the cluster:

As mentioned earlier, the Redis cluster does not use a proxy at the client level, and no matter which node in the cluster is accessed by the Redis client, it can be routed to the corresponding node. Let’s see how the Redis client calls the cache node through routing. of:

(1) MOVED request:

As shown in the figure above, the Redis client calculates the value of Slot through CRC16(key)%16383, and finds that it needs to find "Cache Node 1" for data operations, but the data of this corresponding Slot is migrated due to cache data migration or other reasons Go to "Cache Node 2". Then the Redis client cannot get data from "Cache Node 1" at this time. But because "Cache Node 1" saves the information of all cache nodes in the cluster, it knows that the data of this Slot is saved in "Cache Node 2", so it sends a MOVED redirect request to the Redis client. This request tells it the address of "Cache Node 2" that it should visit. The Redis client gets this address, continues to visit "Cache Node 2" and gets the data.

(2) ASK request:

The above example shows that the data slot has been migrated from "Cache Node 1" to "Cache Node 2", then the client can directly find "Cache Node 2" for data. So if two cache nodes are doing node data migration, how will client requests be processed at this time?

The Redis client sends a request to "Cache Node 1". At this time, "Cache Node 1" is migrating data to "Cache Node 2". If it does not hit the corresponding Slot, it will return an ASK redirect request to the client and tell the "Cache The address of node 2". The client sends an Asking command to "Cache Node 2" and asks whether the required data is on "Cache Node 2". After receiving the message, "Cache Node 2" returns the result of whether the data exists.

(3) Dealing with network overhead caused by frequent redirection: smart client

① What is a smart client:

In most cases, there may be a request redirection to find the correct node. This redirection process will obviously increase the network burden of the cluster and the time consumption of a single request. So most of the clients are smart. The so-called smart client means that the client locally maintains a mapping table cache of hashslot => node. In most cases, you can find hashslot => node directly by going to the local cache. There is no need for moved redirection through the node.

② The working principle of JedisCluster:

When JedisCluster is initialized, a node is randomly selected, hashslot => node mapping table is initialized, and a JedisPool connection pool is created for each node.

Every time an operation is performed based on JedisCluster, the hashslot of the key is first calculated locally, and then the corresponding node node is found in the local mapping table.

If the node still holds the hashslot, then it is ok; if the reshard operation is performed, the hashslot may no longer be on the node, and it will return moved.

If the JedisCluter API finds that the corresponding node returns moved, use the metadata returned by the node to update the local hashslot => node mapping table cache

Repeat the above steps until you find the corresponding node. If you retry more than 5 times, then an error will be reported, JedisClusterMaxRedirectionException

③ Hashslot migration and ask redirection:

If the hashslot is migrating, it will return ask and redirect to the client. After the client receives the ask redirect, it will relocate to the target node for execution, but because the ask occurs during the hashslot migration process, the JedisCluster API will not update the hashslot local cache when the ask is received.

Although both ASK and MOVED control the redirection of the client, there are essential differences. ASK redirection indicates that the cluster is migrating slot data, and the client cannot know when the migration is completed, so it can only be a temporary redirection, and the client will not update the slots cache. But MOVED redirection means that the slot corresponding to the key has been explicitly assigned to the new node, and the client needs to update the slots cache.

Three, the communication mechanism of the nodes in the Redis cluster: goosip protocol

The hash slot algorithm of the redis cluster solves the problem of data access. Different hash slots are located on different nodes, and different nodes maintain a copy of the current cluster state it thinks. At the same time, the Redis cluster is going to Centralized architecture. Then, when the state of the cluster changes, such as new node joining, slot migration, node downtime, slave promotion to new master, etc., we hope that these changes will be discovered by other nodes as soon as possible. How does Redis handle it? In other words, how do different Redis nodes communicate to maintain the synchronization state of the cluster?

In a Redis cluster, the gossip protocol is used to communicate between different nodes, and the purpose of communication between nodes is to maintain metadata information between nodes. These metadata are what data each node contains, whether there is a failure, through the gossip protocol, to achieve the consistency of the final data.

The gossip protocol is a protocol for information exchange between nodes or processes based on the spread of epidemics. The principle is to continuously communicate and exchange information between different nodes. After a period of time, all nodes will have complete information of the entire cluster, and the status of all nodes will reach agreement. Each node may know all other nodes, or only a few neighbor nodes, but as long as these nodes can be connected through the network, their state will be consistent in the end. The biggest advantage of the Gossip protocol is that even if the number of cluster nodes increases, the load on each node will not increase much and is almost constant.

The communication process of the nodes in the Redis cluster is as follows:

Each node in the cluster will open a separate TCP channel for communication between nodes.

Each node selects several nodes to send ping messages through pending rules in a fixed period

The node that receives the ping message responds with a pong message

The advantage of using the gossip protocol is to disperse the metadata update on different nodes, reducing the pressure; but the disadvantage is that the metadata update has a delay, which may cause some lag in some operations in the cluster. In addition, because the gossip protocol requires high server time, inaccurate timestamps will affect the validity of the node's judgment of the message. Moreover, the network overhead after the increase in the number of nodes will also put pressure on the server. At the same time, the number of nodes is too large, which means that the time to reach the final consistency is relatively longer. Therefore, the official recommended maximum number of nodes is about 1000.

Each redis under the redis cluster architecture must open two port numbers, for example, one is 6379, and the other is the port number 16379 plus 1w.

The port number 6379 is the entrance of the redis server.

The 16379 port number is used for communication between nodes, that is, the thing of the cluster bus. The communication of the cluster bus is used for failure detection, configuration update, and failover authorization. The cluster bus uses a binary protocol called gossip protocol

1. Common types of gossip protocol:

The common message types of the gossip protocol include: ping, pong, meet, fail, and so on.

(1) meet: Mainly used to notify new nodes to join the cluster. Through the "cluster meet ip port" command, the nodes of the existing cluster will send invitations to the new node to join the existing cluster.

(2) ping: Used to exchange node metadata. Each node sends a ping message to other nodes in the cluster every second. The message encapsulates the status of its own node and the status data of other parts of the node, including the slot information it manages, and so on.

Because some metadata should be carried when sending a ping command, if it is very frequent, it may increase the burden on the network. Therefore, generally each node will perform 10 pings per second, and each time it will select 5 other nodes that have not communicated for the longest time.

If it is found that the communication delay of a certain node has reached cluster_node_timeout / 2, then send the ping immediately to avoid excessive data exchange delay and cause serious information delay. For example, if there is no data exchange between two nodes for 10 minutes, then the entire cluster is in a serious metadata inconsistency, and there will be a problem. So cluster_node_timeout can be adjusted. If it is adjusted larger, the frequency of ping will be reduced.

Each time you ping, you will bring the information of your own node, and also bring 1/10 of the information of other nodes, send it out, and exchange it. It contains at least 3 other nodes' information and at most (total number of nodes-2) other nodes' information.

(3) pong: The response to ping and meet messages also contains the status of its own node and cluster metadata information.

(4) Fail: After a node judges that another node has failed, it broadcasts the message that the node is down to all nodes in the cluster, and other nodes mark that they are offline after receiving the message.

Due to the decentralization of the Redis cluster and the gossip communication mechanism, the nodes in the Redis cluster can only guarantee final consistency. For example, when joining a new node (meet), only the inviting node and the invited node know about this, and the rest of the nodes have to wait for the ping message to spread layer by layer. Except for Fail, which is notified immediately by the entire network, other nodes such as new nodes, re-online of nodes, election of slave nodes to become master nodes, slot changes, etc., all need to wait to be notified, that is, the Gossip protocol is an eventually consistent protocol.

2. Implementation of the meet command:

(1) Node A will create a clusterNode structure for node B and add the structure to its clusterState.nodes dictionary.

(2) Node A sends a MEET message to node B according to the IP address and port number given by the CLUSTER MEET command.

(3) Node B receives the MEET message sent by node A, node B will create a clusterNode structure for node A, and add the structure to its clusterState.nodes dictionary.

(4) Node B returns a PONG message to node A.

(5) Node A will receive the PONG message returned by node B. Through this PONG message, node A can know that node B has successfully received the MEET message sent by itself.

(6) After that, node A will return a PING message to node B.

(7) Node B will receive the PING message returned by node A. Through this PING message, node B can know that node A has successfully received the PONG message returned by itself, and the handshake is complete.

(8) After that, node A will propagate the information of node B to other nodes in the cluster through the Gossip protocol, so that other nodes will also shake hands with node B. Finally, after a period of time, node B will be used by all nodes in the cluster. understanding.

Fourth, the expansion and contraction of the cluster:

As a cache node for distributed deployment, it will always encounter the problems of cache expansion and cache failure. This will cause the online and offline problems of the cache node. Since slot data is stored in each node, when the number of cache nodes changes, these slot data will be migrated to other cache nodes according to the corresponding virtual slot algorithm. So for a redis cluster, cluster scaling mainly lies in the movement of slots and data between nodes.

1. Expansion:

(1) Start a new node

(2) Use the cluster meet command to join the new node to the cluster

(3) Migrating slots and data: After adding a new node, some slots and data need to be migrated from the old node to the new node

As shown in the figure above, "Cache Node 1" and "Cache Node 2" originally existed in the cluster. At this time, "Cache Node 3" went online and joined the cluster. At this time, according to the algorithm of the virtual slot, the data of the corresponding slots in "Cache Node 1" and "Cache Node 2" will be migrated to "Cache Node 3" due to the addition of new nodes.

When a new node joins the cluster, as an orphan node, it does not communicate with other nodes. Therefore, it is necessary to execute the cluster meet command on any node in the cluster to let the new node join in. Assuming that the new node is 192.168.1.1 5002 and the old node is 192.168.1.1 5003, then run the following command to add the new node to the cluster.

192.168.1.1 5003> cluster meet 192.168.1.1 5002

This is initiated by the old node, which means that old members welcome new members to join. The new node has just been created without the data corresponding to the slot, which means that no data is cached. If this node is the master node, it needs to expand the slot data; if this node is a slave node, it needs to synchronize the data on the master node. In short, it is to synchronize data.

As shown in the figure above, the client initiates the migration of slot data between nodes, and the data is migrated from the source node to the target node.

(1) The client initiates a command to prepare to import slot data to the target node, so that the target node is ready to import slot data. Here the cluster setslot {slot} importing {sourceNodeId} command is used.

(2) After that, send a command to the source node to prepare the source node to move out of the corresponding slot data. Use the command cluster setslot {slot} importing {sourceNodeId}.

(3) At this time, the source node is ready to migrate the data, and the data to be migrated is obtained before the migration. Through the command cluster getkeysinslot {slot} {count}. Count represents the number of slots to be migrated.

(4) Then execute on the source node, migrate {targetIP} {targetPort} "" 0 {timeout} keys {keys} command to migrate the obtained keys to the target node in batches through the pipeline.

(5) Repeat steps 3 and 4 to continuously migrate data to the target node.

(6) After completing the data migration to the target node, notify the corresponding slot to be allocated to the target node through the cluster setslot {slot} node {targetNodeId} command, and broadcast this information to other master nodes in the entire network, and update its own slot node correspondence table.

2. Shrink:

Migration tank.

Forget about the node. Notify other nodes through the command cluster forget {downNodeId}

In order to safely delete nodes, the Redis cluster can only offline nodes that do not have a responsible slot. Therefore, if you want to go offline with a master node responsible for the slot, you need to first migrate the slot it is responsible for to other nodes. The migration process is also similar to the online operation. The difference is that when you go offline, you need to notify other nodes in the entire network to forget you. At this time, use the command cluster forget {downNodeId} to notify other nodes.

5. The cluster's failure detection and failover recovery mechanism:

1. Cluster failure detection:

Redis cluster failure detection is based on the gossip protocol. Each node in the cluster will periodically send PING messages to other nodes in the cluster to exchange status information of each node and detect the status of each node: online status, suspected offline status PFAIL, FAIL in offline status.

(1) Subjective offline (pfail): When node A detects that the communication time with node B exceeds cluster-node-timeout, it will update the local node status and update node B subjectively offline.

Subjective offline does not mean that a node is actually offline. It may be that the network between node A and node B is disconnected, but other nodes can still communicate with node B.

(2) Objective offline:

Since the nodes in the cluster will continue to communicate with other nodes, and the offline information will be transmitted to all nodes through Gossip messages, the nodes in the cluster will continue to receive offline reports.

When more than half of the master nodes mark that node B is subjectively offline, the objective offline process will be triggered (this process is only for the master node, if it is a slave node, it will be ignored). Save the subjective offline report to the local ClusterNode structure fail_reports linked list, and check the timeliness of the subjective offline report, if it exceeds the time of cluster-node-timeout*2, ignore this report, otherwise record the report Content and mark it as objective offline .

Then broadcast a Fail message from the master node B to the cluster, and all nodes that receive the message will mark node B as objectively offline.

2. Cluster fault recovery:

When the failed node goes offline, if it is the master node holding the slot, it needs to find one of its slave nodes to replace it, so as to ensure high availability. At this time, all slave nodes of the offline master node are responsible for recovery. These slave nodes will regularly monitor whether the master node enters an objective offline state, and if so, trigger the failure recovery process. Failure recovery is to elect a node to act as the new master. The election process is implemented based on the Raft protocol election method.

2.1. Filtering from the node:

Check the time for each slave node to disconnect from the master node. If it exceeds cluster-node-timeout * cluster-slave-validity-factor, then it is not eligible to switch to master

2.2. Voting:

(1) Node ordering:

Sort all the slave nodes that pass the filter conditions, and sort them according to priority, offset, and run id. The higher the ranking, the higher the priority for election.

The lower the value of priority, the higher the priority

The larger the offset, the more data copied from the master node, the earlier the election time, and the election will be given priority

If the offset is the same, the smaller the run id, the higher the priority

(2) Update configuration era:

Each master node will update the configuration epoch (clusterNode.configEpoch), this value is an increasing integer. This value records the version of each node and the version of the entire cluster. Whenever important things happen (for example: new nodes appear, slave nodes are selected), the global configuration epoch will be added and assigned to the relevant master node to record this event. The purpose of updating this value is to ensure that all master nodes are consistent with this "big event", and everyone is unified into a configuration epoch, which means that everyone is aware of this "big event".

(3) Initiate an election:

After updating the configuration epoch, the slave node will initiate a broadcast election message (CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST) to the cluster, requiring all master nodes that have received this message and have voting rights to vote. Each slave node can only initiate one election in an epoch.

(4) Election voting:

If a master node has the right to vote, and the master node has not yet voted for other slave nodes, the master node will return a CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK message to the slave node that requires voting, indicating that the master node supports the slave node to become the new master node. Each slave node participating in the election will receive the CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK message, and count how many master nodes it has received based on how many such messages it has received.

If more than (N/2 + 1) number of master nodes have voted for a certain slave node, then the election is passed, this slave node can be switched to master, if the slave node does not get it within the time of cluster-node-timeout*2 With a sufficient number of votes, the election is invalidated, the configuration epoch is updated, and the second round of elections is conducted until a new master node is elected.

In step (1), the leading slave node usually gets more votes, because it triggers the election earlier and has a greater chance of getting votes

2.3. Replace the master node:

When the slave node that meets the voting conditions is selected, the operation of replacing the master node will be triggered. Delete the slot data that the original master node is responsible for, add these slot data to its own node, and broadcast to let other nodes know about this, and a new master node is born.

(1) The selected slave node executes the SLAVEOF NO ONE command to make it the new master node

(2) The new master node will revoke all slot assignments to the offline master node and assign all these slots to itself

(3) The new master node broadcasts PONG messages to the cluster to inform other nodes that they have become the new master node

(4) The new master node starts to receive and process slot-related requests

Note: If the master and slave nodes of a node in the cluster are both down, the cluster will enter the fail state because the slot mapping of the cluster is incomplete. If more than half of the masters in the cluster hang up, regardless of whether there is a slave, the cluster will enter the fail state.

Six, Redis cluster construction:

This part can refer to this article: https://juejin.cn/post/6922690589347545102#heading-1

The establishment of a Redis cluster can be divided into the following parts:

1. Start the node: Start the node in cluster mode, read or generate the cluster configuration file, and the node is independent at this time.

2. Node handshake: The nodes communicate through the gossip protocol to connect independent nodes into a network, mainly using the meet command.

3. Slot assignment: 16384 slots are allocated to the master node to achieve the effect of saving database key-value pairs in shards.

Seven, Redis cluster operation and maintenance:

1. Data migration issues:

Redis clusters can dynamically expand and shrink nodes. This process is currently in a semi-automatic state and requires manual intervention. When scaling up or down, data migration is required. In Redis, in order to ensure the consistency of migration, all migration operations are synchronous operations. When the migration is performed, Redis at both ends will enter a blocking state of varying duration. For small keys, the time can be ignored, but if the memory of the key is If it is used too large, it will contact the failover within the cluster in severe cases, causing unnecessary switching.

2. Bandwidth consumption problem:

Redis cluster is a cluster architecture without a central node. It relies on the Gossip protocol to cooperate and automatically repair the state of the cluster, but goosip has the problem of message delay and message redundancy. When there are too many cluster nodes, goosip protocol communication will consume a lot of bandwidth. , Mainly reflected in the following aspects:

Message sending frequency: closely related to cluster-node-timeout, when a node finds that the last communication time with other nodes exceeds cluster-node-timeout/2, it will directly send a ping message
Message data volume: The main data occupancy of each message includes: slots array (2kb) and 1/10 of the status data of the entire cluster
Machine scale for node deployment: The upper limit of machine bandwidth is fixed. Therefore, the more machines distributed in a cluster of the same scale, the more even the nodes divided by each machine, the higher the overall available bandwidth in the entire cluster

Cluster bandwidth consumption is mainly divided into: read and write command consumption + Gossip message consumption, so building a Redis cluster needs to make a reasonable plan according to the business data scale and message communication cost:

Try to avoid large clusters while meeting business needs. The same system can split and use several clusters for different business scenarios.
Moderately provide cluster-node-timeout to reduce the frequency of message sending, but cluster-node-timeout also affects the speed of failover, so you need to balance the two according to your own business scenarios
If conditions permit, try to deploy them evenly on more machines and avoid centralized deployment. If a cluster of 60 nodes is deployed on 3 machines with 20 nodes each, the bandwidth consumption of the machines will be very serious

3. Pub/Sub broadcast problem:

In cluster mode, all publish commands will be broadcast to all nodes internally, which increases the bandwidth burden, so the cluster should avoid frequent use of Pub/sub functions

4. Cluster tilt:

Cluster tilt refers to the obvious difference in data volume and request volume between different nodes. This situation will increase the difficulty of load balancing and development, operation and maintenance. Therefore, it is necessary to understand the reasons for cluster tilt

(1) Data tilt:

Uneven distribution of nodes and slots
The number of keys corresponding to different slots is too different
Collection object contains a large number of elements
Memory-related configuration is inconsistent

(2) Request tilt:

Design keys reasonably, split large collections of hot objects or use hmget instead of hgetall to avoid overall reading

5. Cluster read-write separation:

The cost of read-write separation in cluster mode is relatively high. It is a better choice to directly expand the number of master nodes to improve cluster performance.

Reference article: https://baijiahao.baidu.com/s?id=1663270958212268352&wfr=spider&for=pc