Due to the large amount of data , a single Master replica set is difficult to bear. Therefore, multiple replica sets need to be clustered to form a horizontal expansion. Each replica set is only responsible for storing a part of the entire data set. This is the Redis cluster, and its role is to provide An assembly for sharing data among multiple Redis nodes.

To sum it up in one sentence: Redis cluster is an assembly that provides shared data between multiple Redis nodes, and Redis cluster can support multiple masters

effect

1. Redis cluster supports multiple Masters, and each Master can mount multiple Slaves

Read-write separation: the master slave model supports data read-write separation

Support high availability of massive data: one master follows multiple slaves, when a master hangs up, its corresponding slave takes over to ensure high availability of data

Support for massive data read and write storage operations: from one master to multiple masters, it has also been greatly optimized in terms of supporting data volume, so the redis cluster supports massive data read and write storage operations

2. Since Cluster comes with Sentinel's failover mechanism, it has built-in high-availability support, so there is no need to use the Sentinel function

The node connection between the client and Redis no longer needs to connect to all nodes in the cluster, just connect to any available node in the cluster

3. Slots are assigned to each physical service node, and the corresponding cluster is responsible for maintaining the relationship between nodes, slots and data

Cluster algorithm-fragmentation-slot slot

The concept of slot and allocation and the advantages of both

Official website introduction analysis

The cluster's key space is divided into 16384 slots, effectively setting an upper cluster size limit of 16384 master nodes (however, the recommended maximum node size is around 1000 nodes).

Each master node in the cluster handles a subset of the 16384 hash slots. When no cluster reconfiguration is in progress (i.e. hash slots are moved from one node to another), the cluster is stable. When the cluster is stable, a single hash slot will be served by a single node (however, a serving node can have one or more replicas, replacing it in case of a network partition or failure, and can be used to scale out. Reading stale data is acceptable operation).

slot

The Redis cluster does not use a consistent hash, but introduces the concept of hash slots. The Redis cluster has 16384 hash slots. Each key is moduloed by CRC16 to determine which slot to place in. Each node in the cluster is responsible for Part of the hash slot, for example, if there are three nodes in the following figure, then:

Fragmentation

What is sharding	When using Redis cluster, we will distribute the stored data to multiple redis machines, which is called sharding. In short, each Redis instance in the cluster is considered a shard of the entire data.
How to find the shard for a given key	To find the shard for a given key, we apply the CRC16(key) algorithm to the key and modulo the total number of shards. Then, using a deterministic hash function , which means that a given key will always map to the same shard multiple times , We can infer where to read a particular key in the future.

Advantages of both

The biggest advantage lies in scaling and data fragmentation search

This structure is easy to add or delete nodes. For example, if I want to add a new node D, I need to get some slots from nodes A, B, and C to D. If I want to remove node A, I need to add Move the slots to B and C nodes and then remove the A node without any slots from the cluster. Since moving the hash slot from one node to another does not stop the service, no matter whether you add, delete or change The number of hash slots on a node will not cause the cluster to be unavailable.

Three solutions for slot slot mapping

hash remainder partition

200 million records are 200 million k, v, we must distribute multiple machines if we can’t stand alone, assuming that there are 3 machines to form a cluster,

Each read and write operation of the user is based on the formula:

hash(key) % N number of machines, calculate the hash value to determine which node the data is mapped to.

advantage:

Simple and crude, direct and effective, you only need to estimate the data and plan the nodes, such as 3, 8, or 10 nodes, to ensure data support for a period of time. Use the Hash algorithm to let a fixed part of the requests fall on the same server, so that each server can process a fixed part of the requests (and maintain the information of these requests),

Play the role of load balancing + divide and conquer.

shortcoming:

It is more troublesome to expand or shrink the originally planned nodes. No matter the expansion or contraction, every time the data changes, the nodes will change.

The mapping relationship needs to be recalculated. There is no problem when the number of servers is fixed. If elastic expansion or failure shutdown is required,

The original modulo formula will change: Hash(key)/3 will become Hash(key)/?.

At this time, the result of the remainder operation of the address will change greatly, and the server obtained according to the formula will also become uncontrollable.

A certain redis machine is down, and due to the change in the number of machines, it will cause all the hash data to be reshuffled.

Consistent Hash Algorithm Partitioning

Three major steps:
1. Algorithm to build a consistent hash ring

consistent hash ring

A consistent hash algorithm must have a hash function and generate a hash value according to the algorithm. All possible hash values of this algorithm will form a full set. This set can become a hash space [0,2^32-1], which is A linear space, but in the algorithm, we connect it end to end (0 = 2^32) through appropriate logic control, so that it logically forms a circular space.

It is also based on the method of using the modulus. The node modulo method introduced in the previous notes is to take the modulus of the number of nodes (servers) . The consistent Hash algorithm is to take the modulus of 2^32 . Simply put, the consistent Hash algorithm organizes the entire hash value space into a virtual ring . For example, suppose the value space of a certain hash function H is 0-2^ 32-1 (that is, the hash value is a 32-bit unsigned integer), the entire hash ring is as follows: the entire space is organized in a clockwise direction , and the point directly above the ring represents 0, the first one on the right of point 0 The point represents 1, and so on, 2, 3, 4, ... until 2^32-1, that is to say, the first point on the left of point 0 represents 2^32-1, and 0 and 2^32-1 are in The directions in the zero point coincide, and we call this ring composed of 2^32 points a Hash ring.

2. redis server ip node mapping

node mapping

Map each IP node in the cluster to a certain position on the ring.

Use Hash to perform a hash on each server. Specifically, you can choose the IP or host name of the server as a keyword for hashing, so that each machine can determine its position on the hash ring. If the four nodes NodeA, B, C, and D are calculated by the hash function of the IP address (hash(ip)), the positions in the ring space after using the IP address hash are as follows:

3. Key drop rules for key falling to the server

When we need to store a kv key-value pair, first calculate the hash value of the key, hash(key), use the same function Hash to calculate the hash value of this key and determine the position of this data on the ring, from this position along the ring "Walking" clockwise , the first server encountered is the server it should locate, and the key-value pair is stored on this node.

For example, we have four data objects, Object A, Object B, Object C, and Object D. After hash calculation, the positions in the ring space are as follows: According to the consistent Hash algorithm, data A will be assigned to Node A, B is routed to Node B, C is routed to Node C, and D is routed to Node D.

advantage:

fault tolerance

Assuming that Node C is down, you can see that objects A, B, and D will not be affected at this time. Generally, in the consistent Hash algorithm, if a server is unavailable, the affected data is only from this server to the previous server in its ring space (that is, the first server encountered when walking counterclockwise) ) data , others will not be affected. Simply put, when C is down, only the data between B and C will be affected and the data will be transferred to D for storage.

Scalability

The amount of data has increased, and a node NodeX needs to be added. The position of X is between A and B, and the affected data is the data between A and X. Just re-enter the data from A to X to X. ,

It will not result in reshuffle of all data after hashing.

shortcoming:

Data Skew Problem of Hash Ring

When the consistent Hash algorithm has too few service nodes , it is easy to cause data skew due to uneven distribution of nodes (most of the cached objects are cached on a certain server).

For example, there are only two servers in the system:

Summarize:

In order to migrate data as little as possible when the number of nodes changes

Arrange all the storage nodes on the Hash ring that is connected at the end. After calculating the Hash, each key will find the adjacent storage node clockwise for storage.

When a node joins or exits, it only affects the subsequent nodes adjacent to the node clockwise on the Hash ring .

advantage

Adding and deleting nodes only affects the clockwise adjacent nodes in the hash ring, and has no effect on other nodes.

shortcoming

The distribution of data is related to the location of the nodes, because these nodes are not evenly distributed on the hash ring, so the data cannot be evenly distributed when stored.

Hash slot partitioning

Concept: In order to solve the data skew problem of the consistent hash algorithm, the hash slot is essentially an array, and the array [0,2^14 -1] forms the hash slot space.

Function: To solve the problem of uniform distribution, another layer is added between the data and the nodes. This layer is called a hash slot (slot), which is used to manage the relationship between the data and the nodes. Now it is equivalent to putting The slot is the slot, and the data is placed in the slot.

The slot solves the problem of granularity, which is equivalent to increasing the granularity, which facilitates data movement. Hash solves the mapping problem, using the hash value of the key to calculate the slot where it is located, which is convenient for data allocation

How many hash slots:

A cluster can only have 16384 slots, numbered 0-16383 (0-2^14-1). These slots are allocated to all master nodes in the cluster, the allocation policy is not required.

The cluster will record the corresponding relationship between nodes and slots. After solving the relationship between nodes and slots, the next step is to calculate the hash value of the key, and then take the modulus of 16384. The remainder is a few keys and then falls into the corresponding slot. HASH_SLOT = CRC16(key) mod 16384. Data is moved in units of slots, because the number of slots is fixed, and it is easier to handle, so the problem of data movement is solved.

Hash slot calculation:

There are 16384 hash slots built into the Redis cluster, and redis will map the hash slots to different nodes approximately equally according to the number of nodes. When a key-value needs to be placed in the Redis cluster, redis first uses the crc16 algorithm to calculate a result for the key and then uses the result to find the remainder of 16384 [CRC16(key) % 16384], so that each key will correspond to a number between 0- The hash slot between 16383 is mapped to a certain node. In the following code, key A and B are on Node2, and key C is on Node3

classic interview questions

Why is the maximum number of slots in the redis cluster 16384?

(1) If the slot is 65536, the header of the sent heartbeat message reaches 8k, and the sent heartbeat packet is too large.

The most space-consuming in the message header is myslots[CLUSTER_SLOTS/8]. When the slot is 65536, the size of this block is: 65536÷8÷1024=8kb The most occupied space in the message header is myslots[CLUSTER_SLOTS/8]. When the slot is 16384, the size of this block is: 16384÷8÷1024=2kb because every second, the redis node needs to send a certain number of ping messages as heartbeat packets. If the slot is 65536, the header of the ping message Too big and a waste of bandwidth.

(2) The number of redis cluster master nodes is basically impossible to exceed 1000.

The more cluster nodes there are, the more data is carried in the message body of the heartbeat packet. If there are more than 1000 nodes, it will also cause network congestion. Therefore, the author of redis does not recommend that the number of redis cluster nodes exceed 1000. Then, for redis clusters with less than 1000 nodes, 16384 slots are enough. There is no need to expand to 65536.

(3) The smaller the slot, the lower the number of nodes, the higher the compression ratio and the easier the transmission.

In the configuration information of the Redis master node, the hash slot it is responsible for is saved in the form of a bitmap, and the bitmap will be compressed during transmission, but if the fill rate slots / N of the bitmap is very high (N means node number), the bitmap compression rate is very low. If the number of nodes is small and the number of hash slots is large, the compression rate of the bitmap is very low.

Cluster environment case steps

3 master 3 slave redis cluster configuration

Find three virtual machines and create new ones

mkdir -p/myredis/cluster

Create 6 independent redis service instances

The content in each configuration file is similar, and you can adjust it yourself. We give the configuration information of a configuration file:

bind 0.0.0.0
daemonize yes
protected-mode no
port 6381
logfile "/myredis/cluster/cluster6381.log"
pidfile /myredis/cluster6381.pid
dir /myredis/cluster
dbfilename dump6381.rdb
appendonly yes
appendfilename "appendonly6381.aof"
requirepass 111111
masterauth 111111
 
cluster-enabled yes
cluster-config-file nodes-6381.conf
cluster-node-timeout 5000

Start six redis instances:

redis-server/myredis/cluster/redisCluster6381.conf
........
redis-server/myredis/cluster/redisCluster6386.conf

Build a cluster relationship for six machines named by redis-dir

/Attention, attention, attention to your real IP address //Attention, attention, attention to your real IP address

redis-cli -a 111111 --cluster create --cluster-replicas 1 192.168.111.175:6381 192.168.111.175:6382 192.168.111.172:6383 192.168.111.172:6384 192.168.111.174:6385 192.168.111.174:6386

--cluster-replicas 1 means to create a slave node for each master

The link enters 6381 as an entry point to view and verify the status of the cluster

info replication

cluster info

cluster nodes

3 master 3 read and write from redis cluster

Add two keys to 6381 to see the effect:

We can find that: we may report an error when adding a new key, and we analyze the reason: when we set the new key k1, after the crc16 algorithm and modulo operation, the key is assigned to a branch that does not belong to the current slice, this error will be reported, we want to solve this error, we can use the following command:

redis- cli - a - 111111 p 6381 -c

After using the -c command, the current key will be directly assigned to the corresponding shard and slot after calculation

Use cluster keyslot to view the slot value corresponding to the key

Master-slave fault-tolerant switching forward case

case analysis:

6381(master)-6384(slave)

Stop 6381, 6384 will become master

Starting 6381, 6384 is still the master, does not make the
Redis cluster do not guarantee strong consistency, which means that under certain conditions, the Redis cluster may lose some write request commands received by the system

Because the essence is still sending heartbeat packets, it takes some time to judge whether the machine is down. If the machine is down, the corresponding slave will directly become the master.
If you want the original master to continue to be the master, execute the following command:

CLUSTER FAILOVER // Execute this command under the port number of whoever is in charge

Master-slave expansion case

Create two service instance configuration files of 6387 and 6388 + start (add another virtual machine or directly select one of the three virtual machines)
start two new node instances of 87/88, and they are all
masters

Take one of them as an example to illustrate the configuration information in the configuration file:

bind 0.0.0.0
daemonize yes
protected-mode no
port 6387
logfile "/myredis/cluster/cluster6387.log"
pidfile /myredis/cluster6387.pid
dir /myredis/cluster
dbfilename dump6387.rdb
appendonly yes
appendfilename "appendonly6387.aof"
requirepass 111111
masterauth 111111

cluster-enabled yes
cluster-config-file nodes-6387.conf
cluster-node-timeout 5000

Add the newly added 6387 node as the master to the original cluster
redis-cli -a 123456 --cluster add-node 192.168.230.114:6387 192.168.238.111:6381
to check the cluster status, 6381
redis-cli -a 123456 --cluster check 192.168. 238.111:6381
redistribute the slot number
redis-cli -a 123456 --cluster reshard 192.168.238.111:6381
The redistribution cost is too high, so the first three companies each allocate a part, and allocate 1364 from the three old nodes 6381/6383/6385 A hole, note that this machine has been adjusted here, so I need to separate 4096 from 6381

Check the cluster again
redis-cli -a 123456 --cluster check 192.168.238.111:6381

Assign slave node 6388 to master node 6387 –cluster-master-id followed by the id of 6387
redis-cli -a 123456 --cluster add-node 192.168.238.114:6388 192.168.238.114:6387 --cluster-slave --cluster -master-id b861764cbba16a1b21536a3182349748e56f24cc

View cluster information
redis-cli -a 123456 --cluster check 192.168.238.111:6381

Master-slave scaling case

Purpose: Take 6387 and 6388 offline

1. Cluster check for the first time: check 6388 nodes

2. Delete 6388 nodes

Command: redis-cli -a password --cluster del-node ip: slave port slave 6388 node ID

redis-cli -a 111111 --cluster del-node 192.168.111.174:6388 218e7b8b4f81be54ff173e4776b4f4faaf7c13da

3. Clear the 6387 node slots and assign all nodes to 6381

redis-cli -a 111111 --cluster reshard 192.168.111.175:6381

4. Situation check a second time

redis-cli -a 111111 --cluster check 192.168.111.175:6381

All 4096 slots are assigned to 6381, and it becomes 8192 slots, which is equivalent to giving all of them to 6381, otherwise you have to input 3 times, one pot

5. Delete the 6387 node

Command: redis-cli -a password --cluster del-node ip: port 6387 node ID

redis-cli -a 111111 --cluster del-node 192.168.111.174:6387 4feb6a7ee0ed2b39ff86474cf4189ab2a554a40f

6. Last condition check

redis-cli -a 111111 --cluster check 192.168.111.175:6381

Cluster common operation commands and CRC algorithm analysis

Algorithm overview and source code discussion:
The redis cluster has 16384 hash slots. After each key passes the CRC16 check, it takes a modulo of 16384 to determine which slot to place it in. Each node of Jiequn is responsible for a part of the hash slot.

Key-values that are not in the same slot cannot use multi-key operations such as mset and mget. You
can use {} to define the concept of the same group, so that key-value pairs with the same content in {} in the key can be placed in a slot. , compared to the figure below, similar to k1k2k3 are mapped to x, and the natural slots are the same

mset k1{x} v1 k2{x} v2 K3{x} v3
mget k1{x} k2{x} K3{x}

CLUSTER COUNTKEYSINSLOT slot number // 1, the slot is occupied, 0, the slot is not occupied
CLUSTER KEYSLOT key name // Which slot should the key exist in

cluster nodes // View the master-slave relationship of the cluster cluster
info / / View cluster information
info replication // View the master-slave
cluster keyslot k2 // View the hash slot location of the key

[redis] redis cluster

concept