1. Comparison of Redis cluster solutions
Sentinel mode
In versions prior to redis3.0, clusters are generally implemented by using the sentinel tool to monitor the status of the master node. If the master node is abnormal, it will switch between master and slave, and use a slave as the master. The configuration of the sentinel is slightly complicated and its performance It has a general performance in all aspects such as high availability, especially when there is a transient interruption of access at the moment of master-slave switching , and the sentinel mode has only one master node to provide external services, which cannot support high concurrency, and the memory of a single master node is not suitable. If the setting is too large, it will cause the persistent file to be too large and affect the efficiency of data recovery or master-slave synchronization.
High-availability cluster mode
The redis cluster is a distributed server cluster composed of multiple master-slave node clusters . It has the characteristics of replication, high availability, and fragmentation . Redis cluster does not need sentinel to complete node removal and failover functions. Each node needs to be set to cluster mode, this mode is not the center of the cluster nodes, horizontally scalable, according to the official text file called linearly extended to tens of thousands of nodes (the official recommended no more than 1000 nodes). The performance and high availability of the redis cluster are better than the previous version of the sentinel mode, and the cluster configuration is very simple.
Two, Redis high-availability cluster construction
2.1. Install stand-alone redis
The purpose is to get relevant executable commands and original configuration files, refer to: Redis 5.0.0 installation (stand-alone version)
2.2. Install cluster redis
The redis cluster needs at least three master nodes (to ensure that the election of the master meets most of the principles, note: only the master node has the right to vote, the principle is explained below ), we build three master nodes here, and build a slave node for each master There are a total of 6 redis nodes. Here, three machines are used to deploy six redis instances, each machine has one master and one slave. The steps to build a cluster are as follows:
Step 1: Create a folder redis-cluster under /usr/local on the first machine, then create two folders under it, and copy the executable commands to the cluster directory.
（1）mkdir -p /usr/local/redis-cluster
（2）mkdir 9001、 mkdir 9004
(3) cp /usr/local/redis/bin -r /usr/local/redis-cluster (copy the executable command to the cluster directory)
Step 2: Copy the previous original configuration file redis.conf to 9001, and modify the following content:
（1）daemonize yes
（2）port 9001（分别对每个机器的端口号进行设置）
（3）dir /usr/local/redis-cluster/9001/data/（指定数据文件存放位置，必须要指定不同的目录位置，不然会丢失数据）
（4）cluster-enabled yes（启动集群模式）
（5）cluster-config-file nodes-9001.conf（集群节点信息文件，这里900x最好和port对应上）
（6）cluster-node-timeout 5000
（7）# bind 127.0.0.1（去掉bind绑定访问ip信息）
（8）protected-mode no （关闭保护模式）
（9）appendonly yes
# 如果要设置密码需要增加如下配置：
（10）requirepass smsp (设置redis访问密码)
（11）masterauth smsp (设置集群节点间访问密码，跟上面一致)Step 3: Copy the modified configuration file to 9002, modify the port numbers in items 2, 3, and 5, you can replace them in batches:
:%s/source string/destination string/g
Step 4: The other two machines also need to do the above steps, the second machine uses 9002 and 9005, and the third machine uses 9003 and 9006
Step 5: Start 6 redis instances respectively, and then check whether the startup is successful
（1）cd /usr/local/redis-cluster
（2）bin/redis-server 900*/redis.conf
(3) ps -ef | grep redis to see if the startup is successful
Step 6: Use redis-cli to create the entire redis cluster (the previous version of redis5 cluster is implemented by the ruby script redis-trib.rb)
（1）cd /usr/local/redis-cluster
（2）bin/redis-cli -a smsp --cluster create --cluster-replicas 1 10.18.4.25:9001 10.18.4.26:9002 10.18.4.27:9003 10.18.4.25:9004 10.18.4.26:9005 10.18.4.27:9006
--cluster-replicas 1 represents creating a slave server node for each master server node created
Note: When executing --cluster create to create a cluster, the following situations may occur:
[root@smsp-dev001-10 redis-cluster]# bin/redis-cli -a smsp --cluster create --cluster-replicas 1 10.18.4.25:9001 10.18.4.26:9002 10.18.4.27:9003 10.18.4.25:9004 10.18.4.26:9005 10.18.4.27:9006
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.18.4.26:9005 to 10.18.4.25:9001
Adding replica 10.18.4.25:9004 to 10.18.4.26:9002
Adding replica 10.18.4.27:9006 to 10.18.4.27:9003
>>> Trying to optimize slaves allocation for anti-affinity
[OK] Perfect anti-affinity obtained!
M: 2349fa527eebba23a4c2749c56c9ee5a8b0d1812 10.18.4.25:9001
slots:[0-5460] (5461 slots) master
M: 763dfcc1f6bc3a613dc56e77b09b787544297b99 10.18.4.26:9002
slots:[5461-10922] (5462 slots) master
M: bc03f9c9c6fba1ac80f7073a46956cbbcf4b48dc 10.18.4.27:9003
slots:[10923-16383] (5461 slots) master
S: 5c31db9af7f63807fb06c17e8707a0486eee88c5 10.18.4.25:9004
replicates 763dfcc1f6bc3a613dc56e77b09b787544297b99
S: 2251b04ada543220f2e485a7371160fa2da6a059 10.18.4.26:9005
replicates bc03f9c9c6fba1ac80f7073a46956cbbcf4b48dc
S: 95fe85fcf724be804d1ff4a3769ddeffa9f7c4ad 10.18.4.27:9006
replicates 2349fa527eebba23a4c2749c56c9ee5a8b0d1812
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
# 在此等待n久集群节点还是没有成功加入集群，而节点启动端口号也已经在防火墙中开放了
Waiting for the cluster to join
.....................................................................................
.....................................................................................
......................................................After waiting for a long time, the cluster node still failed to join the cluster, and the node startup port number has been opened in the firewall!
Reason: Waiting for the cluster to join redis cluster has been waiting
Solution: Open the start port + 10000 port in the firewall too!
At the same time, each time the cluster is created unsuccessfully, cluster-related files will be generated in the node data directory. Before creating the cluster again, you need to shut down the node first, and then delete all the files in the data directory to recreate the cluster!
Step 7: Verify the cluster:
(1) Just connect to any client: ./redis-cli -c -h -p (-a access server password, -c means cluster mode, specify ip address and port number)
Such as: bin/redis-cli -a smsp -c -h 10.18.4.25 -p 9001
(2) Perform verification: cluster info (view cluster information), cluster nodes (view node list)
(3) Perform data operation verification
(4) To shut down the cluster, you need to shut down one by one, use the command: bin/redis-cli -a smsp -c -h 10.18.4.25 -p 900* shutdown
2.3. Java operation redis cluster
With the help of redis's java client jedis, the above clusters can be operated. The maven coordinates of the jedis version are referenced as follows:
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.9.0</version>
</dependency>It is very simple to write the code to access the redis cluster in Java, as shown below:
package com.ceair.screen.flight.service;
import redis.clients.jedis.HostAndPort;
import redis.clients.jedis.JedisCluster;
import redis.clients.jedis.JedisPoolConfig;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;
/**
* 访问redis集群
* @author smsp
*
*/
public class RedisCluster
{
public static void main(String[] args) throws IOException
{
Set<HostAndPort> jedisClusterNode = new HashSet<HostAndPort>();
jedisClusterNode.add(new HostAndPort("10.18.4.25", 9001));
jedisClusterNode.add(new HostAndPort("10.18.4.26", 9002));
jedisClusterNode.add(new HostAndPort("10.18.4.27", 9003));
jedisClusterNode.add(new HostAndPort("10.18.4.25", 9004));
jedisClusterNode.add(new HostAndPort("10.18.4.26", 9005));
jedisClusterNode.add(new HostAndPort("10.18.4.27", 9006));
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxTotal(100);
config.setMaxIdle(10);
config.setTestOnBorrow(true);
//connectionTimeout：指的是连接一个url的连接等待时间
//soTimeout：指的是连接上一个url，获取response的返回等待时间
JedisCluster jedisCluster = new JedisCluster(jedisClusterNode, 6000, 5000, 10, "smsp", config);
System.out.println(jedisCluster.set("student", "zhuge"));
System.out.println(jedisCluster.set("age", "19"));
System.out.println(jedisCluster.get("student"));
System.out.println(jedisCluster.get("age"));
jedisCluster.close();
}
}The operation effect is as follows:
OK OK smsp 19
3. Analysis of Redis cluster principle
Redis Cluster divides all data into 16384 slots (slots), and each node is responsible for some of the slots. The slot information is stored in each node.
When the client of Redis Cluster connects to the cluster, it will also get a copy of the slot configuration information of the cluster and cache it locally on the client. In this way, when the client wants to find a key, it can directly locate the target node. At the same time, because the slot information may be inconsistent between the client and the server (the corresponding information of the slot and the node will change when the capacity is expanded and contracted), a correction mechanism is also needed to realize the verification and adjustment of the slot information.
Slot location algorithm
By default, Cluster will hash the key value using the crc16 algorithm to get an integer value, and then use this integer value to modulo 16384 to get the specific slot.
HASH_SLOT = CRC16(key) mod 16384
Jump relocation
When the client sends an instruction to a wrong node, the node will find that the slot where the key of the instruction is located is not managed by itself. At this time, it will send a special jump instruction to the client to carry the node address of the target operation. Tell the client to connect to this node to get data. After the client receives the instruction, in addition to jumping to the correct node to operate, it will also update and correct the local slot mapping table cache synchronously, and all subsequent keys will use the new slot mapping table.
Network jitter
Real-world computer room networks are often not calm, and they often have various small problems. For example, network jitter is a very common phenomenon. Suddenly, some connections become inaccessible, and then return to normal soon.
To solve this problem, Redis Cluster provides an option cluster-node-timeout , which means that when a node is out of connection for a continuous timeout, it can be determined that the node is faulty and requires a master-slave switch. Without this option, network jitter will cause frequent master-slave switching (re-duplication of data).