环境：
centos 7.3
ip:10.36.8.106(这里是单台测试，所以没有用bind，直接是127.0.0.1，如果是实际环境得加bind绑定自己的地址)
安装：
yum -y install redis redis-trib
配置：
mkdir -p /opt/redis-cluster/700{0,1,2,3,4,5}
cd /opt/redis-cluster/7000
vim redis.conf
port 7000
daemonize yes
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
#配置复制及修改所有节点
for i in `seq 1 5`;do cp redis.conf ../700$i/redis.conf;sed -i "s/7000/700$i/" ../700$i/redis.conf;done
测试：
cd /opt/redis-cluster
#因为配置文件没有加后台机器，这样批量启动会卡住
for dir in `ls -r`;do cd /opt/redis-cluster/$dir;redis-server redis.conf;done
#加入后台以守护进程方式在批量启动
for dir in `ls -r`;do sed -i "/port/a\daemonize yes" $dir/redis.conf;done
创建集群：
[root@host-192-168-1-100 7000]# redis-trib create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
Can I set the above configuration? (type 'yes' to accept): y
这里必须输入yes，否则会失败
[root@host-192-168-1-100 7000]# redis-trib create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
slots:0-5460 (5461 slots) master
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join...
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
slots:0-5460 (5461 slots) master
1 additional replica(s)
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
1 additional replica(s)
M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
slots: (0 slots) slave
replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots: (0 slots) slave
replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
查看集群状态：(也可以 redis-trib check 127.0.0.1:7000)
[root@host-192-168-1-100 7000]# redis-cli -p 7000 cluster nodes
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524020822039 2 connected 5461-10922
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 master - 0 1524020823042 3 connected 10923-16383
bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000 myself,master - 0 0 1 connected 0-5460
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 slave bc37b785100a2fe0b4575c977cb587908b15f2d6 0 1524020822540 4 connected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524020823543 5 connected
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 slave 575f24b2fe784e2b6d18591e90ff7e760dc272df 0 1524020821537 6 connected
验证添加数据：
必须加-c 以集群模式进入，否则添加key会报错(从7000进入，根据算法直接保存在了7002)
[root@host-192-168-1-100 ~]# redis-cli -p 7000
127.0.0.1:7000>
127.0.0.1:7000>
127.0.0.1:7000> set foo bar
(error) MOVED 12182 127.0.0.1:7002
[root@host-192-168-1-100 ~]# redis-cli -c -p 7000
127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
127.0.0.1:7002>
Redis 集群没有使用一致性hash, 而是引入了 哈希槽的概念.
Redis 集群有16384个哈希槽,每个key通过CRC16校验后对16384取模来决定放置哪个槽.集群的每个节点负责一部分hash槽,
主动主从切换：(进入需要切换的从)
redis-cli -p 7005
cluster failover
主动完成主从切换，不会丢失数据
添加新节点：
新启动的服务加入集群一个主节点的从节点(把maser-id及后面的node-id去掉随机分配一个主，如果--slave也去掉就是新添加一个主节点)
redis-trib add-node --slave --master-id 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7006 127.0.0.1:7000
从更换主节点：
已是从节点可以主动改变自己归属的主节点
进入从节点 (网上说主节点必须空，特意测试发现不用为空也是可以切换的）
redis-cli -c -p 7006
cluster replicate 00a1138c9e4358f5bbd734c6f194bed04a37e999
插入数据：
for i in `seq 1000`;do redis-cli -c -p 7001 set foot$i $i ;done
再次切换
[root@host-192-168-1-100 7007]# redis-cli -c -p 7007
127.0.0.1:7007>
127.0.0.1:7007> cluster replicate 0b1580a5305041bde69108e438653e1bfccefed0
OK
可以切换成功，对应的主id已经变换为切换后的主
[root@host-192-168-1-100 7007]#
[root@host-192-168-1-100 7007]# redis-cli -c -p 7000 cluster nodes
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111358911 8 connected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111357909 2 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524111359413 8 connected
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master - 0 1524111359413 8 connected 10923-16383
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 0 0 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524111357909 7 connected 0-5460
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111358409 8 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524111359413 2 connected 5461-10922
并且切换到新的主下后，与新主的数据是一致的
redis-cli -c -p 7005 keys "*" |awk '{print $0}' |sort -n |uniq -c
*** 通过这种连接方式去get指定值，会关联所有master做查询，任何一个节点存在都会返回，set 设置特定值也是根据CRC16校验后对16384取模来决定放置哪个槽
移除集群重新加入：
移出集群会直接把服务干掉(主如果非空不行，必须得先reshard，从可以干掉)
redis-trib del-node 127.0.0.1:7000 "455a070d53f205d89de8c252ff61997b04852976"
重新启动加入之前的集群时，由于之前产生了集群的唯一id节点信息，所以得干掉node.conf文件在重启
未删掉node.conf时直接加入报错
redis-trib add-node --slave 127.0.0.1:7007 127.0.0.1:7000
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 127.0.0.1:7005
[ERR] Node 127.0.0.1:7007 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.
停止删除node.conf后启动在加入成功
netstat -tnlp |grep 7007 |awk '{print $NF}'|cut -d"/" -f1|sort -n |uniq
kill -INT 30523
rm -f nodes.conf
redis-server redis.conf
redis-trib add-node --slave 127.0.0.1:7007 127.0.0.1:7000
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
Automatically selected master 127.0.0.1:7005
>>> Send CLUSTER MEET to node 127.0.0.1:7007 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 127.0.0.1:7005.
[OK] New node added correctly.
[root@host-192-168-1-100 7007]# redis-cli -c -p 7000 cluster nodes
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524110722826 8 connected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524110724330 2 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524110723829 8 connected
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master - 0 1524110722826 8 connected 10923-16383
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 0 0 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524110723328 7 connected 0-5460
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524110722325 8 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524110723829 2 connected 5461-10922
故障转移或者升级可以先把从切换为主在对应的从上cluster failover
扩容-添加节点：(启动新节点7008,添加add-node不在说明，这里以master身份加入)
[root@host-192-168-1-100 7008]# redis-server redis.conf
[root@host-192-168-1-100 7008]# redis-cli -p 7008 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb :7008 myself,master - 0 0 0 connected
[root@host-192-168-1-100 7008]#
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118789706 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524118788703 2 connected 5461-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118788201 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524118788201 7 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118789205 8 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524118790206 8 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524118789205 7 connected 0-5460
[root@host-192-168-1-100 7008]#
[root@host-192-168-1-100 7008]# redis-trib add-node 127.0.0.1:7008 127.0.0.1:7005
>>> Adding node 127.0.0.1:7008 to cluster 127.0.0.1:7005
>>> Performing Cluster Check (using node 127.0.0.1:7005)
M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
3 additional replica(s)
S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
slots: (0 slots) slave
replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots: (0 slots) slave
replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 127.0.0.1:7008 to make it join the cluster.
[OK] New node added correctly.
[root@host-192-168-1-100 7008]#
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524118842819 0 connected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118842319 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524118843820 2 connected 5461-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118843320 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524118842318 7 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118843320 8 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524118842319 8 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524118844322 7 connected 0-5460
上述只是添加到集群成功，但是还没有分配槽不能正常工作
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524119218246 9 connected 0-999
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119219249 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524119219249 2 connected 5461-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119220249 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524119219749 7 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524119218247 9 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524119218747 8 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524119219749 7 connected 1000-5460
[root@host-192-168-1-100 7008]# redis-trib reshard 127.0.0.1:7005 #分配slot
>>> Performing Cluster Check (using node 127.0.0.1:7005)
M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
slots:0-999 (1000 slots) master
1 additional replica(s)
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
2 additional replica(s)
S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
slots: (0 slots) slave
replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
slots: (0 slots) slave
replicates 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots: (0 slots) slave
replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
slots:1000-5460 (4461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 20 #选择分配多少数量的slot
What is the receiving node ID? 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb #上述数量的slot分配给谁
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all #all代表从所有的master里面根据算法选择上述数量slot槽给receiving node ID
Ready to move 20 slots. #done代表指定从某个master选择上述slot给receiving node ID
Source nodes:
M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots:10923-16383 (5461 slots) master
1 additional replica(s)
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5461-10922 (5462 slots) master
2 additional replica(s)
M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
slots:1000-5460 (4461 slots) master
1 additional replica(s)
Destination node:
M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
slots:0-999 (1000 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 5461 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5462 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5463 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5464 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5465 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5466 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5467 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 5468 from 0b1580a5305041bde69108e438653e1bfccefed0
Moving slot 10923 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10924 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10925 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10926 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10927 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10928 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 10929 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
Moving slot 1000 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
Moving slot 1001 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
Moving slot 1002 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
Moving slot 1003 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
Moving slot 1004 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
Do you want to proceed with the proposed reshard plan (yes/no)?
Moving slot 5461 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5462 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5463 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5464 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5465 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5466 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5467 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 5468 from 127.0.0.1:7001 to 127.0.0.1:7008:
Moving slot 10923 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10924 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10925 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10926 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10927 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10928 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 10929 from 127.0.0.1:7005 to 127.0.0.1:7008:
Moving slot 1000 from 127.0.0.1:7003 to 127.0.0.1:7008:
Moving slot 1001 from 127.0.0.1:7003 to 127.0.0.1:7008:
Moving slot 1002 from 127.0.0.1:7003 to 127.0.0.1:7008:
Moving slot 1003 from 127.0.0.1:7003 to 127.0.0.1:7008:
Moving slot 1004 from 127.0.0.1:7003 to 127.0.0.1:7008:
那如果你先选择从某一个指定节点分配到另外一个节点的话
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
在这里先填入指定节点回车在输入done回车即可
第一次转移1000选择的是done，从7003转移;后续转移20,从7003转移5个，7001转移8个：5461-5468，7005转移7个：10923-10929
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524119466369 9 connected 0-1004 5461-5468 10923-10929
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119465867 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524119466871 2 connected 5469-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10930-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119466870 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524119467371 7 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524119467873 9 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524119467873 8 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524119466369 7 connected 1005-5460
缩容-缩减节点：
前提：必须得先转移master节点上所分配的所有槽以及转移所有的从节点(如果从节点也干掉的话就先reshard master,在干掉这个这个master和slave)
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524120043364 9 connected 5461-5468 10923-10929
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120044363 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120044363 2 connected 5469-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10930-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120045367 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120043864 10 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524120044865 9 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120044865 8 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120043364 10 connected 0-5460
[root@host-192-168-1-100 7008]#
上面采取reshard+done 指定转移的方式把1005 转移给了7003，测试删除一下发现报错，非空的，必须得想Ianreshard data
[root@host-192-168-1-100 7008]# redis-trib del-node 127.0.0.1:7000 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
>>> Removing node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb from cluster 127.0.0.1:7000
[ERR] Node 127.0.0.1:7008 is not empty! Reshard data away and try again.
[root@host-192-168-1-100 7008]# redis-trib reshard 127.0.0.1:7005
>>> Performing Cluster Check (using node 127.0.0.1:7005)
M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots:10930-16383 (5454 slots) master
1 additional replica(s)
M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
slots:5461-5468,10923-10929 (15 slots) master
1 additional replica(s)
S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
slots:5469-10922 (5454 slots) master
2 additional replica(s)
S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
slots: (0 slots) slave
replicates 0b1580a5305041bde69108e438653e1bfccefed0
S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
slots: (0 slots) slave
replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
slots: (0 slots) slave
replicates 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
slots: (0 slots) slave
replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
slots:0-5460 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 15
What is the receiving node ID? 00a1138c9e4358f5bbd734c6f194bed04a37e999
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Source node #2:done
Ready to move 15 slots.
Source nodes:
M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
slots:5461-5468,10923-10929 (15 slots) master
1 additional replica(s)
Destination node:
M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
slots:10930-16383 (5454 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 5461 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5462 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5463 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5464 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5465 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5466 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5467 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 5468 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10923 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10924 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10925 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10926 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10927 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10928 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Moving slot 10929 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 5461 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5462 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5463 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5464 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5465 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5466 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5467 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 5468 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10923 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10924 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10925 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10926 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10927 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10928 from 127.0.0.1:7008 to 127.0.0.1:7005:
Moving slot 10929 from 127.0.0.1:7008 to 127.0.0.1:7005:
再次reshard之后，查看集群情况7008已经是空的，测试删除成功(上述缩减reshard的时候注意只能分配给指定的）
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524120244373 9 connected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120246378 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120245376 2 connected 5469-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 11 connected 5461-5468 10923-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120245877 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120246378 10 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120244875 11 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120246378 11 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120244875 10 connected 0-5460
[root@host-192-168-1-100 7008]#
[root@host-192-168-1-100 7008]# redis-trib del-node 127.0.0.1:7000 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
>>> Removing node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb from cluster 127.0.0.1:7000
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
bc37b785100a2fe0b4575c977cb587908b15f2d6 :0 master,fail,noaddr - 1524025875825 1524025874721 1 disconnected
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120264923 5 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120265424 2 connected 5469-10922
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 11 connected 5461-5468 10923-16383
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120266929 8 connected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120265927 10 connected
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120264923 11 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120266427 11 connected
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120266929 10 connected 0-5460
前面说了主从主动切换cluster failover，现在是集群自动熔断：
为了方便拍错，记录日志
cd /opt/redis-cluster
for i in `ls -r`;do echo "logfile \"/var/log/redis/$i.log\"" >> /opt/redis-cluster/$i/redis.conf;done
重启服务
pkill redis-server
for i in `ls -r`;do cd /opt/redis-cluster/$i;redis-server redis.conf;done
现在down掉主：
当都是1主1从时，从直接使用failover选举，并宣告自己是主
当down的是1主1从，其他是1主多从时，选举这个从为主之后，其它多从的会选择一个从来成为这个新主的附属
[root@host-192-168-1-100 ~]# redis-cli -p 7000 cluster nodes
3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 slave 790b56a8774e24455f6da822aae1c06c8898d45b 0 1524125881979 13 connected
575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 master - 0 1524125880975 14 connected 5461-5468 10923-16383
02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524125880975 2 connected
22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524125881979 8 connected
0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524125880473 2 connected 5469-10922
018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 575f24b2fe784e2b6d18591e90ff7e760dc272df 0 1524125880473 14 connected
00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master,fail - 1524125827918 1524125826314 11 disconnected
790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,master - 0 0 13 connected 0-5460
31105:S 19 Apr 16:17:07.887 # Connection with master lost.
31105:S 19 Apr 16:17:07.888 * Caching the disconnected master state.
31105:S 19 Apr 16:17:08.021 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:08.021 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:08.021 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:09.022 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:09.022 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:09.022 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:10.025 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:10.026 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:10.026 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:11.028 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:11.028 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:11.028 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:12.031 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:12.031 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:12.031 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:13.033 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:13.033 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:13.033 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:13.337 * FAIL message received from 790b56a8774e24455f6da822aae1c06c8898d45b about 00a1138c9e4358f5bbd734c6f194bed04a37e999
31105:S 19 Apr 16:17:13.337 # Cluster state changed: fail
31105:S 19 Apr 16:17:13.434 # Start of election delayed for 827 milliseconds (rank #0, offset 3389).
31105:S 19 Apr 16:17:14.035 * Connecting to MASTER 127.0.0.1:7005
31105:S 19 Apr 16:17:14.035 * MASTER <-> SLAVE sync started
31105:S 19 Apr 16:17:14.035 # Error condition on socket for SYNC: Connection refused
31105:S 19 Apr 16:17:14.209 * Ignoring FAIL message from unknown node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb about 00a1138c9e4358f5bbd734c6f194bed04a37e999
31105:S 19 Apr 16:17:14.335 # Starting a failover election for epoch 14.
31105:S 19 Apr 16:17:14.366 # Failover election won: I'm the new master.
31105:S 19 Apr 16:17:14.366 # configEpoch set to 14 after successful failover
31105:M 19 Apr 16:17:14.366 * Discarding previously cached master state.
31105:M 19 Apr 16:17:14.367 # Cluster state changed: ok
31105:M 19 Apr 16:17:19.946 * Slave 127.0.0.1:7006 asks for synchronization
31105:M 19 Apr 16:17:19.946 * Full resync requested by slave 127.0.0.1:7006 #这个是从其他一主多从的有一个重新请求做为这个新主的从
31105:M 19 Apr 16:17:19.946 * Starting BGSAVE for SYNC with target: disk
31105:M 19 Apr 16:17:19.949 * Background saving started by pid 31269
31269:C 19 Apr 16:17:19.965 * DB saved on disk
31269:C 19 Apr 16:17:19.966 * RDB: 2 MB of memory used by copy-on-write
31105:M 19 Apr 16:17:20.063 * Background saving terminated with success
31105:M 19 Apr 16:17:20.063 * Synchronization with slave 127.0.0.1:7006 succeeded
31113:M 19 Apr 16:17:13.335 * Marking node 00a1138c9e4358f5bbd734c6f194bed04a37e999 as failing (quorum reached).
31113:M 19 Apr 16:17:13.335 # Cluster state changed: fail
31113:M 19 Apr 16:17:14.208 * Ignoring FAIL message from unknown node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb about 00a1138c9e4358f5bbd734c6f194bed04a37e999
31113:M 19 Apr 16:17:14.366 # Failover auth granted to 575f24b2fe784e2b6d18591e90ff7e760dc272df for epoch 14
31113:M 19 Apr 16:17:14.406 # Cluster state changed: ok
31113:M 19 Apr 16:17:19.443 # Connection with slave 127.0.0.1:7006 lost.
31324:M 19 Apr 16:31:00.714 * Clear FAIL state for node 0b1580a5305041bde69108e438653e1bfccefed0: master without slots is reachable again.
31324:M 19 Apr 16:31:01.670 * Slave 127.0.0.1:7001 asks for synchronization
31324:M 19 Apr 16:31:01.670 * Full resync requested by slave 127.0.0.1:7001
31324:M 19 Apr 16:31:01.670 * Starting BGSAVE for SYNC with target: disk
31324:M 19 Apr 16:31:01.672 * Background saving started by pid 31375
31375:C 19 Apr 16:31:01.690 * DB saved on disk
31375:C 19 Apr 16:31:01.691 * RDB: 2 MB of memory used by copy-on-write
31324:M 19 Apr 16:31:01.715 * Background saving terminated with success
31324:M 19 Apr 16:31:01.715 * Synchronization with slave 127.0.0.1:7001 succeeded
Redis 集群的一致性保证(Redis Cluster consistency guarantees)
Redis 集群不保证强一致性。实践中，这意味着在特定的条件下，Redis 集群可能会丢掉一些被系统收 到的写入请求命令。
Redis 集群为什么会丢失写请求的第一个原因，是因为采用了异步复制。这意味着在写期间下面的事情 发生了：
你的客户端向主服务器 B 写入。
主服务器 B 回复 OK 给你的客户端。
主服务器 B 传播写入操作到其从服务器 B1，B2 和 B3。
第二种就是用CLUSTER FAILOVER 命令 这里需要登录从然后输入命令 就能把对应的主换成从了，这种方法不会丢数据
redis-cluster选举:
(1)领着选举过程是集群中所有master参与,如果半数以上master节点与master节点通信超过(cluster-node-timeout),认为当前master节点挂掉.
(2):什么时候整个集群不可用(cluster_state:fail),当集群不可用时,所有对集群的操作做都不可用，收到((error) CLUSTERDOWN The cluster is down)错误
a:如果集群任意master挂掉,且当前master没有slave.集群进入fail状态,也可以理解成进群的slot映射[0-16383]不完成时进入fail状态.
b:如果进群超过半数以上master挂掉，无论是否有slave集群进入fail状态.