Redis cluster is a distributed database solution provided by Redis. The cluster uses sharding to share data, and provides replication and failover operations.
A Redis cluster usually consists of multiple nodes. At the beginning, each node is independent of each other. They are in a cluster that only contains themselves. We use the CLUSTER MEET command to connect the nodes together to form a multiple The cluster of nodes.
The data structure of the cluster:
The clusterNode structure saves the current state of a node, such as the node creation time, node name, the current configuration epoch of the node, and the node's ip port. Each node uses a clusterNode structure to record its own state, and creates a corresponding clusterNode structure for all other nodes in the cluster.
struct clusterNode{
//创建节点的时间
mstime_t ctime;
//节点的名称，由40个十六进制字符组成
char name[REDIS_C:USTER_NAMELEN]
//节点标识（标识节点的角色以及节点目前状态）
inf flags;
//节点当前的配置纪元
uint64_t configEpochl;
//节点的ip地址
char ip;
//节点的端口号
int port;
//保存连接节点所需要的有关信息
clusterLink *link;
}
typedef struct clusterLink{
//连接的创建时间
mstime_t ctime;
//TCP 套接字描述
int fd;
//输出缓冲区，保存着等待发送给其他节点的消息
sds sndbuf;
//输入缓冲区，保存着从其他节点接收到的消息
sds rcvbuf;
//与这个连接相关联的节点，如果没有的话为NULL
struct clusterNode *node;
} clusterLink;
Each node saves a clusterState structure, which records the current state of the cluster from the perspective of the current node.
typedef struct clusterState{
//指向当前节点的指针
clusterNode *myself;
//集群当前的配置纪元，用于实现故障转移。
uint64_t currentEpoch;
//集群当前的状态：在线或下线
int state;
//集群中至少处理着一个槽的节点数量
int size;
//集群节点名单（包括myself节点）
//字典的键为节点的名称，字典值为节点对应的clusterNode结构
dict *node;
} clusterState;
Slot assignment:
The Redis cluster saves the key-value pairs in the database by sharding: the entire database of the cluster is divided into 16,348 slots, each key in the database belongs to one of the 16384 slots, and each node in the cluster can handle 0 Up to 16384 slots.
After using the cluster meet command to connect the nodes to the cluster, the cluster is still offline because the nodes in the cluster have not processed any slots
You can assign slots to nodes by using the cluster addslots <slot> command
Record the slot assignment information of the node:
The slots attribute and numslot attribute of the clusterNode structure record which slots the node is responsible for processing:
struct clusterNode{
unsigned char slots[16348/8];
int numslots;
};
At the same time, the node will send its own slots array to other nodes in the cluster through messages to inform other nodes that they are currently responsible for processing those slots.
The slots array in the clusterState structure records the assignment information of all 16384 slots in the cluster.
typedef struct clusterState{
clusterNode *slots[16384];
}clusterState;
clusterState.slots is to locate the node O(i) where the slot is located faster.
clusterNode.slots When the program needs to send the slot assignment information of a node to other nodes through a message, the program only needs to send the entire clusterNode.slots array of the corresponding node. clusterState.slots records all the slot assignments in the cluster. The clusterNode.slots only records the slot assignment information of the current node.
When the client sends a command related to the database key to the node, the node receiving the command will calculate which slot the database key to be processed by the command belongs to, and check whether this slot is assigned to itself:
If the slot where the key is located is assigned to the current node, the node directly executes this command; if the slot where the key is located is not assigned to the current node, the node will return a MOVED error to the client, directing the client to the correct node , And send the command you wanted to execute again.
The node uses the following algorithm to calculate which slot a given key belongs to:
def slot_number(key):
return CRC16(key) & 16383
When the node calculates the slot i to which the key belongs, the node checks its item i in the clusterState.slots array to determine whether the slot is responsible for it: if clusterState.slots[i] is equal to clusterState.myself, then the slot i is the responsibility of the current node, and the node can execute the command sent by the client; otherwise, the node will return the MOVED error to the client according to the node IP and port number recorded in the clusterNode structure pointed to by clusterState.slots[i] Process the node of slot i.
The format of MOVED errors is: MOVED <slot> <ip >:< port>
When the client receives the MOVED error returned by the node, the client turns to the node responsible for processing the slot according to the IP address and port number provided by the MOVED error, and resends the command it wants to execute before to the node. A cluster client usually creates a socket connection with multiple nodes in the cluster, and the so-called node redirection is actually to change a socket to send commands.
When the cluster mode redis-cli client receives a MOVED error, it will not print the MOVED error. Instead, it will automatically turn the node based on the MOVED error and print the steering information, so we cannot see the MOVED error returned by the node.
When there is a difference between the node and the stand-alone server in terms of database, the node can only use the 0 database, while the stand-alone Redis server does not have this restriction. In addition to storing key-value pairs in the database, the node also uses the slots_to_keys jump table in the clusterState structure to save the relationship between slots and keys:
typedef struct clusterState{
zskiplist *slots_to_keys;
} clusterState;
The score of each node in the slots_to_keys hopping table is a slot number, and the member of each node is a database key: whenever a node adds a new key-value pair to the database, the node will This key and the key slot number are associated with the slots_to_keys jump table; when the node deletes each key-value pair in the database, the node will disassociate the deleted key from the slot number in the slots_to_keys jump table.
By recording the slot to which each database key belongs in the slots_to_keys jump table, the node can easily perform batch operations on all database keys belonging to a certain slot or certain slots.
The resharding operation of the Redis cluster can change any number of slots that have been assigned to a node (source node) to another node, and the key-value pairs belonging to the relevant slot will also be moved from the source node to the target node. The re-dispatch operation can be performed online. During the re-sharding process, the cluster does not need to go offline, and both the source node and the target node can continue to process command requests.
The redis cluster re-sharding operation is performed by Redis's cluster management software redis-trib. Redis provides all the commands needed for re-sharding, and redis-trib sends commands to the source node and target node to perform the re-sharding. Fragmentation operation.
1) redis-trib sends the CLUSTER SETSLOT <slot> IMPORTING <source_id> command to the target node to prepare the target node to import key-value pairs belonging to the slot from the source node.
2) Redis-trib commands the CLUSTER SETSLOT< slot> MIGATING <target_id> command to prepare the source node to migrate the key-value pairs belonging to the slot to the target node.
3) redis-trib sends CLUSTER GETKEYSINGSLOT <slot> <count> to the source node to obtain up to count key-value pairs belonging to the slot.
4) For the key name obtained in step 3, redis-trib sends a MIGRATE <target_ip> <target_port> <key_name> 0 <timeout> command to the source node to atomically migrate the selected key from the source node to the target node.
5) Repeat steps 3 and 4 until all key-value pairs are migrated to the target node.
6) redis-trib sends the CLUSTER SETSLOT <slot> NODE <target_id> command to any node in the cluster to assign the slot to the target node. This assignment information is sent to the entire cluster through a message, and finally all nodes in the cluster will Until the slot slot has been assigned to the target node.
When the client sends a command related to the database key to the source node, and the database key to be processed by the command happens to belong to the slot being migrated: the source node will first search for the specified key in its own database, and if found, Directly execute the command sent by the client; on the contrary, if the source node fails to find the specified key in its own database, then the key may have been migrated to the target node, and the source node returns an ASK error to the client to guide the client The end turns to the target node that is importing the slot, and sends the command that it wants to execute again.
The importing_slots_from array of the clusterState structure records the slots that the current node is importing from other nodes:
typedef struct clusterState{
clusterNode *importing_slots_from[16384];
} clusterState;
If the value of importing_slots_from[i] is not NULL, but points to a clusterNode structure, then it indicates that the current node is importing slot i from the node identified by clusterNode
The clusterState structure migrating_slots_to array records the slots of the current node that is being migrated to other nodes:
typedef struct clusterState{
clusterNode *migratubg_slots_to[16384];
}clusterState;
If the value of migrating_slots_to[i] is not NULL, but points to a clusterNode structure, it means that the current node is migrating slot i to the node identified by clusterNode.
The difference between ASK error and MOVED error:
MOVED error means that the responsibility of the slot has been transferred from one node to another node: after the client receives the MOVED error of slot i, every time the client encounters a command request for slot i, it can directly send the command request To the node pointed to by the MOVED error, because this node is the node currently responsible for slot i.
The ASK error is just a temporary measure used by the two nodes in the process of migrating slots. The wrong direction of ASK will not have any impact on the client's future command requests for slot i. The client will still send the command request for slot i to the node currently responsible for processing slot i.
Learn a little every day, there will always be gains.
Note: Respect the author's intellectual property rights, refer to "Redis Design and Implementation" for the content in the article, and only learn here to share with you.
