Redis master-slave replication, sentry, cluster

Redis Cluster

1. Redis cluster mode

The redis cluster has three modes, namely master-slave synchronization/replication, sentinel mode, and Cluster. The following will explain the working methods of the three modes and how to build a cluster cluster

• Master-slave replication: Master-slave replication is the basis of highly available Redis. Sentinels and clusters are based on master-slave replication to achieve high availability. Master-slave replication mainly implements multi-machine backup of data, as well as load balancing for read operations and simple fault recovery.
  Defects: failure recovery cannot be automated; write operations cannot be load-balanced; storage capacity is limited by a single machine.
• Sentinel: Based on master-slave replication, Sentinel implements automatic failure recovery.
  Defects: write operations cannot be load-balanced; storage capacity is limited by a single machine; Sentinel cannot automatically failover slave nodes. In the scenario of read-write separation, slave node failures will cause read services to be unavailable, and additional monitoring of slave nodes is required , Switch operation.
• Cluster: Through clustering, Redis solves the problem that write operations cannot be load-balanced, and the storage capacity is limited by a single machine, and realizes a relatively complete high-availability solution.

---

2. Redis master-slave replication

2.1 Introduction

Master-slave replication refers to copying the data of one Redis server to other Redis servers. The former is called the master node (Master), and the latter is called the slave node (Slave); data replication is one-way, only from the master node to the slave node.

By default, each Redis server is a master node; and a master node can have multiple slave nodes (or no slave nodes), but a slave node can only have one master node.

2.2 Function

• Data redundancy: master-slave replication implements hot backup of data, which is a data redundancy method other than persistence.
• Fault recovery: When there is a problem with the master node, the slave node can provide services to achieve rapid fault recovery; it is actually a kind of service redundancy.
• Load balancing: On the basis of master-slave replication, combined with read-write separation, the master node can provide write services, and the slave nodes can provide read services (that is, the application connects to the master node when writing Redis data, and the application connects to the slave node when reading Redis data) , to share the server load; especially in the scenario of writing less and reading more, sharing the read load through multiple slave nodes can greatly increase the concurrency of the Redis server.
• The cornerstone of high availability: In addition to the above functions, master-slave replication is also the basis for the implementation of sentinels and clusters, so master-slave replication is the basis for high availability of Redis

2.3 Master-slave replication process

1. If a Slave machine process is started, it will send a "sync command" command to the Master machine to request a synchronous connection.
2. Regardless of whether it is the first connection or reconnection, the Master machine will start a background process to save the data snapshot to the data file (perform rdb operation), and the Master will also record all commands to modify the data and cache them in the data file.
3. After the background process completes the cache operation, the Master machine will send the data file to the Slave machine, and the Slave machine will save the data file to the hard disk, and then load it into the memory, and then the Master machine will combine all operations that modify the data Send to the Slave end machine. If the Slave fails and causes a downtime, it will automatically reconnect after returning to normal.
4. After the Master machine receives the connection from the Slave machine, it sends its complete data file to the Slave machine. If the Mater receives synchronization requests from multiple Slaves at the same time, the Master will start a process in the background to save the data file. Then send it to all Slave-side machines to ensure that all Slave-side machines are normal.

2.4 Build master-slave replication

Install the Redis service
and configure the same for the three servers

[root@lwb ~]# systemctl stop firewalld
[root@lwb ~]# setenforce 0
[root@lwb ~]# yum install -y gcc-c++ make

[root@lwb ~]# cd /opt
[root@lwb opt]# tar xf redis-5.0.7.tar.gz
[root@lwb opt]# cd redis-5.0.7/
[root@lwb redis-5.0.7]# make prefix=/usr/local/redis install

[root@lwb redis-5.0.7]# cd utils
[root@lwb utils]# ./install_server.sh

[root@lwb utils]# ln -s /usr/local/redis/bin/* /usr/local/bin

[root@lwb utils]# vim /etc/redis/6379.conf

Write down their respective ip addresses

[root@lwb utils]# /etc/init.d/redis_6379 restart

#可以在全局使用redis
[root@lwb utils]# ln -s /etc/init.d/redis_6379 /usr/local/bin/redis

#进入redis
[root@lwb utils]# redis-cli -h 192.168.36.40 -p 6379

Modify the Redis configuration file (Master node operation)

[root@master ~]# vim /etc/redis/6379.conf
#第70行，修改监听地址
bind 0.0.0.0 

第137行，开启守护进程 
daemonize yes  

#第172行，指定日志文件目录 
logfile /var/log/redis_6379.log

#第264行，指定工作目录
dir /var/lib/redis/6379

#第288行，添加主服务器地址及端口 
replicaof 192.168.48.11 6379

#第700行，开启AOF持久化功能 
appendonly yes

[root@master ~]# /etc/init.d/redis_6379 restart

Modify the Redis configuration file (Slave node operation)

The configuration of the two slaves is the same

[root@slave1 ~]# vim /etc/redis/6379.conf
#第70行，修改监听地址
bind 0.0.0.0 

第137行，开启守护进程 
daemonize yes  

#第172行，指定日志文件目录 
logfile /var/log/redis_6379.log

#第264行，指定工作目录
dir /var/lib/redis/6379

#第287行，添加主服务器地址及端口 
replicaof 192.168.36.30 6379

#第700行，开启AOF持久化功能 
appendonly yes

[root@slave1 ~]# /etc/init.d/redis_6379 restart

Verify master-slave replication

Check the log on the master node: /var/log/redis_6379.log

Synchronized successfully from the server

[root@master ~]# redis-cli -h 192.168.36.30 -p 6379  #登录数据库

info replication  #查看主从同步信息

Create data on the master

View on slave

---

3. Sentry mode

The method of master-slave switching technology is: when the server is down, it is necessary to manually switch a slave machine to the master machine, which requires manual intervention, which is not only time-consuming and laborious, but also causes the service to be unavailable for a period of time. In order to solve the shortcomings of master-slave replication, there is a sentinel mechanism.

Sentinel's core functions:

Based on master-slave replication, Sentinel introduces automatic failover of the master node.

The principle of sentinel mode:

Sentinel is a distributed system used to monitor each server in the master-slave structure. When a failure occurs, a new Master is selected through a voting mechanism and all Slaves are connected to the new Master. So the number of clusters running Sentinels must not be less than 3 nodes.

The role of sentinel mode:

• Monitoring: Sentry constantly checks that the master and slave nodes are functioning properly.
• Automatic failover: When the master node fails to work normally, Sentinel will start an automatic failover operation. It will upgrade one of the slave nodes of the failed master node to a new master node, and let other slave nodes copy the new master node instead.
• Notification (reminder): Sentinel can send the result of failover to the client.

The sentinel structure consists of two parts, sentry nodes and data nodes:

• Sentinel node: The sentinel system consists of one or more sentinel nodes, which are special redis nodes that do not store data.
• Data Nodes: Both master and slave nodes are data nodes.

failover mechanism

1. The sentinel node regularly monitors to find out whether the master node is faulty.
   Each sentinel node will send a ping command to the master node, slave node and other sentinel nodes every 1 second for a heartbeat detection. If the master node does not reply within a certain time frame or replies with an error message, then the sentinel will consider the master node to be offline subjectively (unilaterally). When more than half of the sentinel nodes think that the master node is offline subjectively, it will be objectively offline
2. When the master node fails, the sentinel node will implement the election mechanism through the Raft algorithm (election algorithm) to jointly elect a sentinel node as the leader to be responsible for handling the failover and notification of the master node. So the number of clusters running Sentinels must not be less than 3 nodes.
3. The failover is performed by the leader sentinel node, the process is as follows:
   • Upgrade a slave node to the new master node, and let other slave nodes point to the new master node
   • If the original master node recovers, it becomes a slave node and points to the new master node
   • Notify the client that the primary node has been replaced

It is important to note that objective offline is a concept unique to the master node; if a slave node or sentinel node fails and is subjectively offlined by the sentinel, there will be no subsequent objective offline and failover operations.

Master node election

1. Filter out unhealthy (offline) slave nodes that do not respond to sentinel ping responses.
2. Select the slave node with the highest priority configuration in the configuration file. (replica-priority, default is 100)
3. Select the slave node with the largest replication offset, that is, the most complete replication.

---

4. Sentry mode configuration

The start of sentry depends on the master-slave mode, so the master-slave mode must be installed before doing the sentinel mode

#所有节点配置一样
vim /opt/redis-5.0.7/sentinel.conf
#第17行，取消注释
protected-mode no

#第21行，Redis哨兵默认的监听端口
port 26379

#第26行，指定sentinel为后台启动
daemonize yes

#第36行，指定日志存放路
logfile "/var/log/sentinel.log"

#第65行，指定数据库存放路径
dir "/var/lib/redis/6379"

#第84行
#指定该哨兵节点监控192.168.36.30:6379这个主节点，该主节点的名称是mymaster,最后的2的含义与主节点的故障判定有关:至少需要2个哨兵节点同意，才能判定主节点故障并进行故障转移
sentinel monitor mymaster 192.168.36.30 6379 2  

#第113行，判定服务器down掉的时间周期，默认30000毫秒(30秒)
sentinel down-after-milliseconds mymaster 30000

#第146行，故障节点的最大超时时间为180000 (180秒)
sentinel failover-timeout mymaster 180000

Start sentry mode

#先启动master，再启动slave
cd /opt/redis-5.0.7/
redis-sentinel sentinel.conf &

View sentinel information

redis-cli -p 26379 info sentinel

Fault simulation

View and kill the redis-server of the master node

Validation results

View sentinel information again

summary

Sentinel mode is based on master-slave replication, but master-slave replication cannot automatically recover after a single point of failure, resulting in high availability of services cannot be achieved; Sentinel mode is based on master-slave replication, adding sentinel node detection, when the master goes down, the sentinel node A new master service will be elected through voting to ensure high availability of the service

---

5. Redis cluster mode

Cluster, namely Redis Cluster, is a distributed storage solution introduced by Redis 3.0.

The cluster consists of multiple nodes (Nodes), and Redis data is distributed among these nodes. The nodes in the cluster are divided into master nodes and slave nodes: only the master node is responsible for the maintenance of read and write requests and cluster information, and the slave nodes only replicate the data and status information of the master node.

5.1 The role of the cluster

1) Data partitioning : Data partitioning (or data sharding) is the core function of the cluster.
The cluster disperses data to multiple nodes. On the one hand, it breaks through the limit of Redis single-machine memory size, and the storage capacity is greatly increased. On the other hand, each master node can provide external read and write services, which greatly improves the responsiveness of the cluster.
Redis stand-alone memory size limitation is mentioned in the introduction of persistence and master-slave replication: for example, if the stand-alone memory is too large, the fork operation of bgsave and bgrewriteaof may cause the master process to block, and the master-slave environment may be switched when the host is switched. As a result, the slave node cannot provide services for a long time, and the replication buffer of the master node may overflow during the full replication phase.

2) High availability : the cluster supports master-slave replication and automatic failover of the master node (similar to Sentinel): when any node fails, the cluster can still provide external services.

5.2 Data Fragmentation of Redis Cluster

Redis cluster introduces the concept of hash slots

Redis cluster has 16384 hash slots (numbered 0-16383)

Each node of the cluster is responsible for a portion of hash slots

After each Key passes the CRC16 check, take the remainder of 16384 to determine which hash slot to place. Through this value, find the node corresponding to the corresponding slot, and then directly and automatically jump to the corresponding node for access operations

Take a cluster composed of 3 nodes as an example:
node A contains hash slots from 0 to 5460,
node B contains hash slots from 5461 to 10922,
and node C contains hash slots from 10923 to 16383

5.3 Master-slave replication model of Redis cluster

There are three nodes A, B, and C in the cluster. If node B fails, the entire cluster will be unavailable due to the lack of slots in the range of 5461-10922.
Add a slave node A1, B1, and C1 to each node, and the entire cluster consists of three master nodes and three slave nodes. After node B fails, the cluster elects the master node with B1 to continue serving. When both B and B1 fail, the cluster will be unavailable

---

6. Build Redis cluster mode

Redis cluster generally requires 6 nodes, 3 masters and 3 slaves

Here all nodes are simulated on the same server, distinguished by port numbers, the master node port numbers are: 6001, 6002, 6003, and the corresponding slave node port numbers are 6004, 6005, 6006

[root@master ~]# cd /etc/redis
[root@master redis]# mkdir -p redis-cluster/redis600{1..6}
[root@master redis]# for i in {1..6}
> do
> cp /opt/redis-5.0.7/src/redis-cli /etc/redis/redis-cluster/redis600$i   
> cp /opt/redis-5.0.7/src/redis-server /etc/redis/redis-cluster/redis600$i
> done

All six revisions

[root@master redis]#  cd /etc/redis//redis-cluster/redis6001
[root@master redis6001]# ls
redis-cli  redis.conf  redis-server
[root@master redis6001]# vim redis.conf
#第69行，注释，即监听所有端口
#bind 127.0.0.1 

#第88行，关闭保护模式 
protected-mode no

#第92行，为了区分，将端口更改，6个不能相同
port 6001

#第136行，开启守护进程
daemonize yes

#第699行，开启AOF持久化
appendonly yes

#第832行，开启集群功能
cluster-enabled yes

#第840行，群集名称文件设置
cluster-config-file nodes-6001.conf

#第846行，群集超时时间设置
cluster-node-timeout 15000 

kill redis-server 6379 process

[root@master redis6006]# ps -ef | grep redis
root       1238      1  0 14:49 ?        00:00:06 /usr/local/bin/redis-server 0.0.0.0:6379
[root@master redis6006]# kill -9 1238

Start the redis node

[root@master redis6006]# for m in {1..6};do
> cd /etc/redis/redis-cluster/redis600$m/
> redis-server redis.conf
> done

Start the cluster

[root@master redis6006]# redis-cli --cluster create 127.0.0.1:6001 127.0.0.1:6002 127.0.0.1:6003 127.0.0.1:6004 127.0.0.1:6005 127.0.0.1:6006 --cluster-replicas 1
# -replicas 1   表示每个主节点有1个从节点
#若使用6台服务器，此处节点ip请换为自己真实ip即端口号

cluster test

redis-cli -p 6001 -c
#-c 参数，节点之间可以相互跳转

cluster slots  
#查看节点的哈希槽编号范围

cluster keyslot 键名
#查看键的哈希槽编号

You can see three sets of master-slave

Then create a key value on 6002, check it on 6006, you can see the key, when you check the key value, it will automatically jump to 6002, when you use 6003 to check the key value, it will be empty, but when you use get to check the key value, it will still be Automatically jump to key's hash slot corresponding to node 6002