拓扑结构
| client 192.168.4.50 | |||||
| PORT 6351 | PORT 6352 | PORT 6353 | PORT 6354 | PORT 6355 | PORT 6356 |
| RedisA 4.51 | RedisB 4.52 | RedisC 4.53 | RedisD4.54 | RedisE 4.55 | RedisF 4.56 |
部署集群
在六台redis服务器安装redis服务（51-56同操作）
[root@redisa ~]# yum - y install gcc
[root@redisa ~]# tar -zxvf redi-4.0.8.tar.gz
[root@redisa ~]# cd redis-4.0.8
[root@redisa ~]# make
[root@redisa ~]# make install
[root@redisa ~]# ./utils/install_server.sh
[root@redisa ~]# vim /etc/redis/6379.conf //注，有的可能是redis.conf
70 bind 192.168.4.51
93 port 6351
[root@redisa ~]# vim /etc/init.d/redis_6379 //改脚本文件
...
REDISPORT="6351 -h 192.168.4.51"
[root@redisa ~]# redis-cli -h 192.168.4.51 -p 6351
192.168.4.51:6351> FLUSHALL //清空redis数据，保持一致
OK
192.168.4.51:6351> keys *
(empty list or set)
启动51-56的集群服务(51为例）
[root@redisa ~]# vim /etc/redis/6379.conf
######################### LUA SCRIPTING ################
815 cluster-enabled yes //启用集群
823 cluster-config-file nodes-6351.conf //指定集群信息文件，为了区分这里把节点改成对应端口号
829 cluster-node-timeout 5000 //请求超时5s
[root@redisa ~]# /etc/init.d/redis_6379 restart
Stopping ...
Redis stopped
Starting Redis server...
[root@redisa ~]# netstat -pntul | grep redis-server
tcp 0 0 192.168.4.51:6351 0.0.0.0:* LISTEN 2279/redis-server 1
tcp 0 0 192.168.4.51:16351 0.0.0.0:* LISTEN 2279/redis-server 1
//集群服务默认端口号16351（默认是端口号加10000）
[root@redisa ~]# ls /var/lib/redis/6379/ //查看集群文件
dump.rdb nodes-6353.conf
查看集群状态信息，未创建前的状态如下：
[root@redisa ~]# redis-cli -h 192.168.4.51 -p 6351
192.168.4.51:6351> CLUSTER info //查看集群信息
cluster_state:fail //未集群前的状态
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
192.168.4.51:6351> CLUSTER nodes
394473afaa2d6fae7e0f621e557f0ea8addb4a9b :6351@16351 myself,master - 0 0 0 connected
在管理主机51（可以随机选）上部署ruby脚本运行环境
（号外：Ruby，面向对象的脚本语言，由日本人松本行弘(Yukihiro Matsumoto)开发，遵守GPL协议和Ruby License。由 Ruby 语言本身还发展出了JRuby（Java平台）、IronRuby（.NET平台）等其他平台的 Ruby 语言替代品。因为Perl发音与6月诞生石pearl（珍珠）相同，因此Ruby以7月诞生石ruby（红宝石）命名）
[root@redisa redis-cluster]# yum -y install ruby //安装依赖环境
已安装:
ruby.x86_64 0:2.0.0.648-30.el7
作为依赖被安装:
libyaml.x86_64 0:0.1.4-11.el7_0
ruby-irb.noarch 0:2.0.0.648-30.el7
ruby-libs.x86_64 0:2.0.0.648-30.el7
rubygem-bigdecimal.x86_64 0:1.2.0-30.el7
rubygem-io-console.x86_64 0:0.4.2-30.el7
rubygem-json.x86_64 0:1.7.7-30.el7
rubygem-psych.x86_64 0:2.0.0-30.el7
rubygem-rdoc.noarch 0:4.0.0-30.el7
rubygems.noarch 0:2.0.14.1-30.el7 //这个作为依赖包也会被安装
[root@redisa redis-cluster]# yum -y install ruby-devel-2.0.0.648-30.el7.x86_64.rpm
[root@redisa redis-cluster]# gem install redis-3.2.1.gem
Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
1 gem installed
[root@redisa ~]# cd redis-4.0.8/src
[root@redisa src]# ls
... redis-trib.rb ... //ruby执行命令
[root@redisa src]# mkdir /root/bin
[root@redisa src]# cp redis-trib.rb /root/bin/ //将命令放入在这个目录下，可以在命令行直接输入redis-trip.rb 命令
[root@redisa src]# ls -ld /root/bin/
drwxr-xr-x. 2 root root 27 12月 22 10:37 /root/bin/
创建集群
[root@redisa src]# redis-trib.rb create --replicas 1 192.168.4.51:6351 192.168.4.52:6352 192.168.4.53:6353 192.168.4.54:6354 192.168.4.55:6355 192.168.4.56:6356
//replicas 数字 自动为每一个节点分配一个给定的数值的从库
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
192.168.4.51:6351
192.168.4.52:6352
192.168.4.53:6353
Adding replica 192.168.4.55:6355 to 192.168.4.51:6351
Adding replica 192.168.4.56:6356 to 192.168.4.52:6352
Adding replica 192.168.4.54:6354 to 192.168.4.53:6353
M: 394473afaa2d6fae7e0f621e557f0ea8addb4a9b 192.168.4.51:6351
slots:0-5460 (5461 slots) master
M: c91b387a09ad23f70ba24b7f1668ea1e37b2db63 192.168.4.52:6352
slots:5461-10922,16287 (5463 slots) master //这里是错误，16287槽点不相连
M: e00875106197cdf9e41f5bb00ca0686b212c0290 192.168.4.53:6353
slots:10923-16383 (5461 slots) master
S: 4eb3de604b688d02fb6feeec514b9fda75c3d897 192.168.4.54:6354
replicates e00875106197cdf9e41f5bb00ca0686b212c0290
S: 5417a13f0b845cd68c12f3f9139cfd22d6806a6e 192.168.4.55:6355
replicates 394473afaa2d6fae7e0f621e557f0ea8addb4a9b
S: 32576b4e2fc456f0adb7b2cd80da49e6950576c5 192.168.4.56:6356
replicates c91b387a09ad23f70ba24b7f1668ea1e37b2db63
Can I set the above configuration? (type 'yes' to accept): yes
/usr/local/share/gems/gems/redis-3.2.1/lib/redis/client.rb:113:in `call': ERR Slot 0 is already busy (Redis::CommandError) //报错信息
[root@redisa ~]# rm -rf /var/lib/redis/6379/nodes-6351.conf //需要将每台机器的这个文件删除
[root@redisa ~]# /etc/init.d/redis_6379 restart //重启查看状态
[root@redisa ~]# vim /var/lib/redis/6379/nodes-6351.conf
641146d06001e3b101e6d1a79e5d0fcec1bd3fa3 :0@0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0 //显示为零才正确
[root@redisa ~]# redis-trib.rb create --replicas 1 192.168.4.51:6351 192.168.4.52:6352 192.168.4.53:6353 192.168.4.54:6354 192.168.4.55:6355 192.168.4.56:6356
....
M: 641146d06001e3b101e6d1a79e5d0fcec1bd3fa3 192.168.4.51:6351
slots:0-5460 (5461 slots) master
M: ea1bff01e1cb40cffa751bbe25e0ec39662b5e66 192.168.4.52:6352
slots:5461-10922 (5462 slots) master //这种才为正确
M: 09b2b77ad7b575f6d0feb0ce03553ac044e0866c 192.168.4.53:6353
slots:10923-16383 (5461 slots) master
....
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
....
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered. //完成
[root@redisa ~]# redis-cli -h 192.168.4.51 -p 6351
192.168.4.51:6351> cluster info //查看信息
cluster_state:ok //显示OK
cluster_slots_assigned:16384
cluster_slots_ok:16384
验证集群：（redis-cli -c -h ip 地址 -p 端口 ） // -c选项是链接集群必须的选项
[root@mysql50 ~]# redis-cli -c -h 192.168.4.52 -p 6352
192.168.4.52:6352> set hary 110
-> Redirected to slot [12751] located at 192.168.4.53:6353 //存到了53上
OK
192.168.4.53:6353>
[root@redisc ~]# redis-cli -h 192.168.4.53 -p 6353 //在53上查看
192.168.4.53:6353> get harry //取值需要进入集群
(error) MOVED 890 192.168.4.51:6351
192.168.4.53:6353> keys * //查看值
1) "hary"
[root@mysql54 ~]# redis-cli -h 192.168.4.54 -p 6354 //登录53对应从库54上
192.168.4.54:6354> keys *
1) "hary"
192.168.4.54:6354> get harry /取值需要链接主库
(error) MOVED 890 192.168.4.51:6351
工作原理：
在三台主库上有hash槽：51是0-5460 52 是5161-10922 53是10923-16383 共计16384个
存入数据的时候会通过内部算法（CRC16算法对key值计算），得到一个hash值，然后这个值和hash槽的总数16384取余，得到的值对应哪个redis服务器就存到哪个redis主库服务器！
（ps：CRC是通信领域中用于校验数据传输正确性的最常用机制，也是Hash算法的一个典型应用，Hash一般翻译为“散列”，也可直接音译为“哈希”，就是把任意长度的输入（又叫做预映射，pre-image）通过散列算法变换成固定长度的输出，该输出就是散列值。）
192.168.4.51:6351> set hary 130
-> Redirected to slot [12751] located at 192.168.4.53:6353
OK
192.168.4.52:6352> set hary 110 //可以发现hash值只和变量有关，和值无关
-> Redirected to slot [12751] located at 192.168.4.53:6353
OK
192.168.4.53:6353> set harry 110
-> Redirected to slot [890] located at 192.168.4.51:6351
OK
管理命令
| 命令 | 选项 | ||||
| redis-cli | -h 帮助 | -h IP地址 | -p 端口 | -c 集群模式 | -a 密码 |
| redis-trib.rb 选项 参数 | add-node 添加master主机 | check 检测集群 | reshard 重新分片 |
add-node --slave 添加slave主机 |
del-node 删除主机 |
选举master主机
master宕机后对应的slave自动被选举为master ，重启后会自动配置为当前的master的slave并且同步数据
[root@mysql52 ~]# redis-cli -h 192.168.4.52 -p 6352
192.168.4.52:6352> keys *
1) "sec"
192.168.4.52:6352> shutdown //关闭52主机
[root@mysql50 ~]# redis-cli -c -h 192.168.4.52 -p 6352
192.168.4.52:6352> set hary 110
192.168.4.51:6351> set marry 123
-> Redirected to slot [8237] located at 192.168.4.56:6356 //可以看到数据存到了52的从库56上面，说明56主机已经自动成为主库
OK
[root@mysql56 ~]# redis-cli -h 192.168.4.56 -p 6356
192.168.4.56:6356> info replication
# Replication
role:master //链接到56上可以看到已经是主库了
connected_slaves:0
192.168.4.56:6356> keys *
1) "sec"
2) "marry" //数据已经在56主机上了
[root@mysql52 ~]# /etc/init.d/redis_6379 start
[root@mysql52 ~]# netstat -pantul | grep :6352
tcp 0 0 192.168.4.52:6352 0.0.0.0:* LISTEN 5194/redis-server 1
[root@mysql52 ~]# redis-cli -h 192.168.4.52 -p 6352
192.168.4.52:6352> info replication //查看主从信息
# Replication
role:slave
master_host:192.168.4.56 //可以看到自动称为56的从库了
master_port:6356
192.168.4.52:6352> keys *
1) "sec"
2) "marry" 数据已经同步
管理集群
新建2台虚拟机并运行redis服务（启用集群模式）
| 192.168.4.57 port 6357 redis57 | 192.168.4.58 port 5358 redis58 |
[root@redis57 redis-4.0.8]# netstat -pantul | grep 6357
tcp 0 0 192.168.4.57:6357 0.0.0.0:* LISTEN 5163/redis-server 1
tcp 0 0 192.168.4.57:16357 0.0.0.0:* LISTEN 5163/redis-server 1
[root@redis58 redis-4.0.8]# netstat -pantul | grep 6358
tcp 0 0 192.168.4.58:6358 0.0.0.0:* LISTEN 6857/redis-server 1
tcp 0 0 192.168.4.58:16358 0.0.0.0:* LISTEN 6857/redis-server 1
将57作为master添加到集群51-56中 在管理主机上操作（51上） redis-trib.rb add-node 新的主机 :端口 集群中的任一主机:端口
[root@redisa ~]# redis-trib.rb add-node 192.168.4.57:6357 192.168.4.51:6351
>>> Adding node 192.168.4.57:6357 to cluster 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.4.57:6357 to make it join the cluster.
[OK] New node added correctly.
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351
>>> Performing Cluster Check (using node 192.168.4.51:6351)
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots: (0 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
重新分配hash槽slots {slot *[slɒt] 【计】 槽; 存储槽}
redis-trib.rb reshard 集群内的主机:端口 指定移除hash槽个数 指定接收槽的id 指定移除hash槽的主机ID
注：数据也会随着hash slots的重新分配而分配到对应的主机
[root@redisa ~]# redis-trib.rb reshard 192.168.4.51:6351
...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096（平均分配）
What is the receiving node ID? ca8be3ee5726215b63d2d38513df11ce3732d706
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1:all
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s)
[OK] All nodes agree about slots configuration.
...
[OK] All 16384 slots covered.
[root@redisa ~]# redis-cli -c -h 192.168.4.57 -p 6357
192.168.4.57:6357> keys *
1) "harry"
2) "hello"
3) "ninhao"
将58作为slave主机，添加到集群中并作为57的从库
redis-trib.rb add-node --slave [ --master-id id值 ] ip地址:端口 集群主机:端口 注：如果不指定主节点，会把新的节点随机添加为节点最少的主的从库
[root@redisa ~]# redis-trib.rb add-node --slave 192.168.4.58:6358 192.168.4.51:6351
...
>> Send CLUSTER MEET to node 192.168.4.58:6358 to make it join the cluster.
Waiting for the cluster to join.
>>> Configure node as replica of 192.168.4.57:6357.
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351
...
S: 585408f1e615c16ecb6670341777288e642b1309 192.168.4.58:6358
slots: (0 slots) slave
replicates ca8be3ee5726215b63d2d38513df11ce3732d706
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
1 additional replica(s)
...
[root@redis58 redis-4.0.8]# redis-cli -h 192.168.4.58 -p 6358 //会自动同步数据
192.168.4.58:6358> keys *
1) "hello"
2) "harry"
3) "ninhao"
移除slave节点 ：因为节点没有槽位，可以直接移除
redis-trib.rb del-node 集群ip:端口 主机id值
移除master节点：重新分片释放占用的hash槽 ，移除master主机
redis-trib.rb reshard 集群主机：ip值
redis-trib.rb del-node 集群主机：端口 master主机id值 移出hash槽个数 接收hash槽的主机ID 移除hash槽主机ID
[root@redisa ~]# redis-trib.rb del-node 192.168.4.51:6351 585408f1e615c16ecb6670341777288e642b1309 //移除192.168.4.58节点
>>> Removing node 585408f1e615c16ecb6670341777288e642b1309 from cluster 192.168.4.51:6351
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots:0-1364,5461-6826,10923-12287 (4096 slots) master
0 additional replica(s) //可以看出57已经没有从库
[root@redisa ~]# redis-trib.rb reshard 192.168.4.51:6351
How many slots do you want to move (from 1 to 16384)? 1365 //可以分配任意槽数给任意主机
What is the receiving node ID? 641146d06001e3b101e6d1a79e5d0fcec1bd3fa3
Source node #1:ca8be3ee5726215b63d2d38513df11ce3732d706
Source node #2:done
....
//这里是分配1365给51 1366给56（因为上面宕机了一次52，主库变为了56） 1365给53
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351
...
M: ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357
slots: (0 slots) master //可以看出57已经没有槽了
0 additional replica(s)
...
[root@redisa ~]# redis-trib.rb check 192.168.4.51:6351 //再次查看已经删除了
把修复的redis服务器再添加到集群中
添加redis服务器57 redis57进入redis---cluster reset（重置集群，这步可以省略）---删除/var/lib/redis/6379/nodes-xxx.conf 文件---重启---链接集群服务器，添加57主机----并重新分片（从库，不需要分片）
[root@mysql57 redis-4.0.8]# /etc/init.d/redis_6379 start
Starting Redis server...
[root@mysql57 redis-4.0.8]# redis-cli -h 192.168.4.57 -p 6357
192.168.4.57:6357> CLUSTER RESET //重置集群信息
OK
[root@mysql57 redis-4.0.8]# cat /var/lib/redis/6379/nodes-6357.conf
ca8be3ee5726215b63d2d38513df11ce3732d706 192.168.4.57:6357@16357 myself,master - 0 1545470578840 9 connected /里面的值不为零，需要删除文件在重启
vars currentEpoch 12 lastVoteEpoch 0
[root@mysql57 redis-4.0.8]# rm -rf /var/lib/redis/6379/nodes-6357.conf
[root@mysql57 redis-4.0.8]# /etc/init.d/redis_6379 restart
[root@mysql57 redis-4.0.8]# cat /var/lib/redis/6379/nodes-6357.conf
d38ebde0b68a65e9f8abcacab1ca54548c0ef381 192.168.4.57:6357@16357 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0
[root@redisa ~]# redis-trib.rb add-node 192.168.4.57:6357 192.168.4.51:6351
//再次添加 并重新分片redis集群单点故障——没有从库的主库宕掉，设置一主两从。当然也可以更多，不过这个足矣了，如果两个从都坏了，估计你就离火车票不远了！！！