Jumpserver high-availability cluster deployment: (4) Redis sentinel mode high-availability deployment

Because Jumpserver itself does not support Redis high availability deployment, Redis sentinel mode is used to achieve data synchronization and failover between Redis's own active and standby nodes, but Jumpserver cannot perceive the IP switching of Redis master nodes, so this article adopts Redis sentinel mode + Keepalived floating IP. With the high availability of Redis, the failover IP of the Redis master node is monitored through Keepalived, so that the floating IP always follows the Redis cluster master node. Jumpserver can always connect to the Redis cluster master node by only connecting to the floating IP.

1. Configure the firewall

6379: Redis listening port
26379: Redis-Sentinel listening port

firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.255.200.1/30" port protocol="tcp" port="6379" accept"
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.255.200.1/30" port protocol="tcp" port="26379" accept"

firewall-cmd --reload

2. Install redis

yum install -y redis

# 创建数据保存目录，将数据保存至SSD磁盘
mkdir /ssd/redis
mkdir /ssd/redis-sentinel
chown -R redis:redis /ssd/redis
chown -R redis:redis /ssd/redis-sentinel

3. Modify the redis configuration file

# 以下仅列出需要修改的配置项
vi /etc/redis.conf

# 修改数据保存位置
dir "/ssd/redis"

# 设置密码
requirepass xxxxxxxx

# 主服务器密码，与前面设置的密码一致
masterauth xxxxxxxx

# 指定主服务器，集群主服务器（第一个启动的节点）不需要配置，其它2个从节点需要配置
slaveof 10.255.200.1 6379

4. Modify the redis-sentinel configuration file

# 备份原始配置文件
cp /etc/redis-sentinel.conf /etc/redis-sentinel.conf.bak 

# 以下配置为全部配置项，只保留了必要配置，其它配置项可全部删除或根据需要修改
vi /etc/redis-sentinel.conf

# 监听IP及端口设置
bind 0.0.0.0
protected-mode no
port 26379

# 修改数据保存位置
dir "/ssd/redis-sentinel"

logfile "/var/log/redis/sentinel.log"

# 设置监控的主服务器信息及认证密码，密码需与前面Redis配置的密码一致
sentinel monitor redismaster 10.255.200.1 6379 2
sentinel auth-pass redismaster xxxxxxxx

# 将故障切换时间改为5秒，默认值为30秒
sentinel down-after-milliseconds redismaster 5000
sentinel failover-timeout redismaster 30000
supervised systemd

5. Start redis high availability cluster

# 注意启动的顺序。首先是主节点的Redis服务进程，然后启动从机的Redis服务进程，最后启动3个哨兵的服务进程

systemctl start redis
systemctl status redis
systemctl enable redis

systemctl start redis-sentinel
systemctl status redis-sentinel
systemctl enable redis-sentinel

6. Cluster status confirmation and data synchronization status confirmation

# 查看Redis集群相关状态，xxxxxxxx 为认证密码，应重点关注当前节点的role是否与实际情况一致（1主2从），master_host 是否是主节点IP，master_link_status 是否为 up 。
redis-cli -a xxxxxxxx info replication

# 查看Redis-Sentinel集群相关状态，xxxxxxxx 为认证密码，应重点关注主节点 ip 是否与实际情况一致，flags 标识是否为 master ,主节点正常时 flags不应为s_down / o_down ，num-slaves 从节点数是否为 2 ，num-other-sentinels 其它哨兵节点数是否为 2 ，quorum 认定主节点故障的“法定人数”是否为 2 。
redis-cli -p 26379 -a xxxxxxxx sentinel master redismaster

# 可以通过在主节点写入/删除数据，在从节点验证数据同步是否正常
redis-cli 

# 授权认证
auth xxxxxxxx

# 主节点写入数据
set test hello-world

# 3个节点读取数据，验证是否同步
get test

# 主节点删除数据
del test

# 3个节点再次读取数据，验证是否均己删除
get test

7. Cluster switch verification

# 停止主节点redis服务
systemctl stop redis

# 等待约5秒，在其它节点上验证集群状态
redis-cli -a xxxxxxxx info replication
redis-cli -p 26379 -a xxxxxxxx sentinel master redismaster

# 也可通过日志查看集群切换过程
tail -100 /var/log/redis/sentinel.log

8. Configure keepalived to implement floating IP to switch with redis master node

# redis集群状态检查脚本，需注意脚本字符串比较时回车换行的处理
# 首先检查 redis 服务是否正常运行，再检查当前节点是否为主节点，再检查当前节点是否持有浮动IP
vi /etc/keepalived/check_redis.sh

#!/bin/bash
</dev/tcp/127.0.0.1/6379
if [ $? -eq 0 ]; then
    ROLE=`redis-cli -a xxxxxxxx info replication | grep role`
    MASTER=`echo -e "role:master\r\n"`
    if [ "$ROLE" == "$MASTER" ]; then
                exit 0
    fi

    MASTERIP=`ip add | grep 10.255.200.4 | wc -l`
    if [ $MASTERIP -eq 0 ]; then
                exit 0
        fi
fi
exit 1

# 编写 Redis 集群切换邮件通知脚本，修改通知标题及正文内容
cp /etc/keepalived/email_tengine.sh /etc/keepalived/email_redis.sh 
vi  /etc/keepalived/email_redis.sh 

#!/bin/bash
contact='[email protected]'
notify() {
    mailsubject="[Devops Redis VIP 切换] $(hostname) -> $1"
    mailbody="[$(date +'%F %T')]: Devops Redis VIP 切换, $(hostname) 切换为 $1 ！"
    echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
    notify master
    ;;
backup)
    notify backup
    ;;
fault)
    notify fault
    ;;
*)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
esac

# 修改keepalived配置，添加一个新的VRRP实例，并添加相关的状态检查脚本
# 注意新的VRRP实例名称以及虚拟路由器ID应与之前的Tengine相关配置不同
# 注意 keepalived 3个节点须工作在非抢占模式下，角色均设置为BACKUP，优先级相同。
vi /etc/keepalived/keepalived.conf

vrrp_script chk_redis {
    script "/etc/keepalived/check_redis.sh"
    interval 1
}

vrrp_instance VI_REDIS {
    state BACKUP
    nopreempt
    interface bond-app
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass xxxxxxxx
    }
    virtual_ipaddress {
        10.255.200.4/24
    }
    track_script {
        chk_redis
    }

    notify_master "/etc/keepalived/email_redis.sh master"
    notify_backup "/etc/keepalived/email_redis.sh backup"
    notify_fault "/etc/keepalived/email_redis.sh fault"
}

# 重新启动 keepalived 服务
systemctl restart keepalived
systemctl status keepalived

# 检查浮动IP是否在主节点上
redis-cli -a xxxxxxxx info replication | grep role
ip add

# 停用主节点redis服务，进行集群切换验证，验证浮动IP是否跟随主节点
# 具体操作参考前面“集群切换验证”内容

9. Redis service recovery in extreme cases

Redis sentinel mode is configured to switch between clusters when two sentinels monitor that the main service is not responding, which can ensure the reliability of cluster switching, but in extreme cases, it may happen that 2 of the 3 nodes in the cluster are temporarily unable to provide In the case of services, the number of sentinels in the cluster at this time does not meet the "quorum" of cluster switching. As a result, the role of the Redis node that survives may always be the role of the slave node. At this time, the Redis service needs to be manually restarted to make the role become The master node restores the Redis service.