Redis cluster
Redis Cluster is a distributed solution for Redis, officially launched in version 3.0, which effectively solves the distributed needs of Redis. When encountering bottlenecks such as single-machine memory, concurrency, and traffic, the Cluster architecture solution can be used to achieve load balancing. Previously, there were generally two types of Redis distributed solutions:
1. The advantage of the client partitioning scheme is that the partitioning logic is controllable, but the disadvantage is that you need to deal with issues such as data routing, high availability, and failover yourself.
2. The agent solution has the advantage of simplifying client distributed logic and facilitating upgrade and maintenance, but the disadvantage is that it increases the complexity of architecture deployment and performance loss.
Now the official provides us with a proprietary cluster solution: Redis Cluster, which very elegantly solves the problems of Redis cluster. Therefore, understanding and applying Redis Cluster well will greatly liberate our workload of using distributed Redis.
Cluster pre-knowledge
Data distribution theory
Distributed databases must first solve the problem of mapping the entire data set to multiple nodes according to partitioning rules, that is, dividing the data set into multiple nodes, and each node is responsible for a subset of the overall data.
What needs to be focused on is the data partitioning rules. Common partitioning rules include hash partitioning and sequential partitioning. Hash partitioning has good dispersion, has nothing to do with data distribution business, and cannot be accessed sequentially. Sequential partitioning has a dispersion that is easily skewed, is related to data distribution business, and can be accessed sequentially.
Node takes residual partition
Use specific data, such as Redis keys or user IDs, and then use the formula: hash(key)%N to calculate the hash value based on the number of nodes N to determine which node the data is mapped to. There is a problem with this solution: when the number of nodes changes, such as expanding or shrinking nodes, the data node mapping relationship needs to be recalculated, which will lead to re-migration of data.
The outstanding advantage of this method is its simplicity. It is often used in the database and table subdivision rules of the database. Generally, the pre-partitioning method is used. The number of partitions is planned in advance according to the amount of data, such as dividing it into 512 or 1024 tables to ensure that it can support the future period. time, and then migrate the table to other databases based on load conditions. When expanding the capacity, double the capacity is usually used to avoid all data mapping being disrupted and causing full migration, as shown in Figure 10-2.
Consistent hash partitioning
The implementation idea of Consistent Hash Table (Distributed Hash Table) is to allocate a token to each node in the system, generally ranging from 0 to 23. These tokens form a hash ring. When data reading and writing perform node search operations, the hash value is first calculated based on the key, and then the first token node that is greater than or equal to the hash value is found clockwise. For example:
There are three nodes (Node1, Node2, Node3) and five keys (key1, key2, key3, key4, key5) in the cluster. The routing rules are:
When a node is added to the cluster, for example, when a node Node4 is added between Node2 and Node3, when node key4 is accessed at this time, it cannot be hit in Node4. More generally, the keys between Node2 and Node4 are invalid. , such a failure method is too "concentrated" and "violent", a better method should be "smooth" and "dispersed" failure.
The biggest advantage of this method compared to node remainder is that adding and deleting nodes only affects adjacent nodes in the hash ring and has no impact on other nodes. But there are several problems with consistent hash partitioning:
1. When using a small number of nodes, node changes will affect the data mapping in the hash ring on a large scale, so this method is not suitable for distributed solutions with a small number of data nodes.
2. Adding nodes can only have a better load sharing effect on the next adjacent node. For example, adding node Node4 in the above figure can only share part of the load on Node3, and has basically no load sharing effect on other nodes in the cluster. ;Similarly, deleting a node will cause the load on the next adjacent node to increase, while other nodes cannot effectively share the load pressure.
Because of these shortcomings of consistent hash partitioning, some distributed systems use virtual slots to improve consistent hashing, such as virtual consistent hash partitioning.
Virtual Consistent Hash Partitioning
In order to maintain a dynamic balance between nodes when adding or deleting nodes, each real node is virtualized into several virtual nodes, and then these virtual nodes are randomly mapped to the ring. At this time, each real node is no longer mapped to the ring. The real node is only used to store key-value pairs, and it is responsible for connecting its own set of virtual nodes on the ring. When accessing and routing key-value pairs, they are first routed to the virtual node, and then the virtual node finds the real node.
As shown in the figure below, there are three real nodes: Node1, Node2 and Node3. Each real node virtualizes three virtual nodes: X#V1, X#V2 and X#V3, so that each real node is responsible for the hash space. It is no longer a continuous section, but is scattered throughout the ring, so that local pressure can be balanced to different nodes. The more virtual nodes, the better the dispersion, and the more uniform the load will be in theory.
virtual slot partition
Redis uses virtual slot partitioning, which can be considered a variant of the above virtual consistent hash partitioning. It uses a hash function with good dispersion to map all data into a fixed range of integer sets. The integers are defined as slots. . This range is generally much larger than the number of nodes. For example, the RedisCluster slot range is 0 ~ 16383. Slots are the basic unit of data management and migration within a cluster. The main purpose of using wide-range slots is to facilitate data splitting and cluster expansion. Each node will be responsible for a certain number of slots.
For example, if the cluster has 3 nodes, each node is responsible for approximately 5460 slots on average. Due to the use of high-quality hashing algorithms, the data mapped by each slot is usually relatively even, and the data is evenly divided into 5 nodes for data partitioning. Redis Cluster uses virtual slot partitioning. The following is an introduction to the Redis data partitioning method.
Why is the slot range from 0 to 16383?
Why does the range of slots range from 0 to 16383, which means the number of slots is 16384? The author of redis has an answer on github: why redis-cluster use 16384 slots? · Issue #2576 · redis/redis · GitHub
This means:
In the Redis cluster, after a successful handshake, ping/pong messages will be sent regularly between nodes to exchange data information. The more nodes in the cluster, the larger the content of the message body. For example, the status information of 10 nodes is about 1kb. At the same time, the nodes in the redis cluster are sending ping messages every second. For example, in a Redis cluster with a total number of nodes of 200, the bandwidth occupied by ping/pong messages reaches 25M by default.
So if the slot is 65536, the message header for sending heartbeat information reaches 8K, and the heartbeat packet sent is too large, which wastes bandwidth.
Secondly, the number of redis cluster master nodes is basically impossible to exceed 1,000. The more cluster nodes there are, the more data is carried in the message body of the heartbeat packet. If there are more than 1,000 nodes, it will also cause network congestion. Therefore, the redis author does not recommend that the number of redis cluster nodes exceed 1,000.
Then, for a redis cluster with less than 1,000 nodes, 16,384 slots are enough to ensure that each master has enough slots, and there is no need to expand to 65,536.
Furthermore, in the configuration information of the Redis master node, the hash slot it is responsible for is saved in the form of a bitmap. During the transmission process, the bitmap will be compressed, but if the filling rate of the bitmap slots / N is high (N represents the number of nodes), that is, if the number of nodes is small and the number of hash slots is large, the compression rate of the bitmap will be very low and resources will be wasted.
Therefore, the Redis author decided to take 16384 slots as a better design trade-off.
Redis data partition
Redis Cluser uses virtual slot partitioning. All keys are mapped to integer slots from 0 to 16383 according to the hash function. The calculation formula is: slot=CRC16(key) &16383. Each node is responsible for maintaining part of the slots and the key value data mapped by the slots.
Features of Redis virtual slot partition
1. Decouple the relationship between data and nodes, simplifying the difficulty of node expansion and contraction.
2. The node itself maintains the mapping relationship of the slots and does not require the client or proxy service to maintain slot partition metadata. The interface supports mapping queries between nodes, slots, and keys, and is used in scenarios such as data routing and online scaling.
3. Data partitioning is the core of distributed storage. Understanding and flexibly using data partitioning rules is very helpful for mastering Redis Cluster.
Cluster function limitations
Redis clusters have some functional limitations compared to single machines, which require developers to understand in advance and avoid them when using them. The restrictions are as follows:
1. Key batch operation support is limited. For example, mset and mget currently only support batch operations on keys with the same slot value. Keys mapped to different slot values are not supported because operations such as mget and mget may exist on multiple nodes.
2. Key transaction operation support is limited. Similarly, only multi-key transaction operations on the same node are supported. When multiple keys are distributed on different nodes, the transaction function cannot be used.
3. Key is the minimum granularity of data partitioning, so a large key-value object such as hash, list, etc. cannot be mapped to different nodes.
4. Multiple database spaces are not supported. Redis in stand-alone mode can support 16 databases. In cluster mode, only one database space can be used, namely db 0.
5. The replication structure only supports one layer. The slave node can only replicate the master node, and the nested tree replication structure is not supported.
Build a cluster
After introducing the Redis cluster partitioning rules, let's start building the Redis cluster. There are several ways to build a cluster:
1) Manually build it according to the Redis protocol and use the cluster meet, cluster addslots, and cluster replicate commands.
2) Before 5.0, redis-trib.rb written in ruby language was used, and the ruby language environment needed to be installed before use.
3) Redis 5.0 and later abandoned redis-trib.rb and merged the cluster building function into redis-cli.
Let’s keep it simple and use the third way to build it. There should be at least an odd number of nodes in the cluster, so there are at least three nodes. The official recommendation is the configuration of three masters and three slaves. Let's build a cluster with three masters and three slaves.
Node configuration
We now stipulate that the ports of the master node are 6900, 6901, and 6902, and the ports of the slave node are 6930, 6931, and 6932.
First, you need to configure the conf file of the node. This is relatively unified. The configuration files of all nodes are similar. Let’s take the node with port 6900 as an example:
--javascripttypescriptbashsqljsonhtmlcssccppjavarubypythongorustmarkdown
port 6900
# 这个部分是为了在一台服务上启动多台Redis服务，相关的资源要改
pidfile /var/run/redis_6900.pid
logfile "/home/lijin/redis/redis/log/6900.log"
dir "/hom