High availability solution Keepalived case (keepalived + nginx)

Case

Nginx + Keepalived case

Use Nginx + Keepalived to implement a web service high availability solution

The architecture diagram is as follows:

Insert image description here

Installation and configuration of Nginx

install software

# 安装nginx以及拓展源
yum install epel-release -y
yum install -y nginx

Nginx configure web site

#web1
[root@nginx1 ~]# vi /etc/nginx/conf.d/web.conf 
server{
      
      
    listen 8080;
    root         /usr/share/nginx/html;
    index test.html;
}

[root@nginx1 ~]# echo "<h1>This is web1</h1>"  > /usr/share/nginx/html/test.html

#web2
[root@nginx2 ~]# vi /etc/nginx/conf.d/web.conf 
server{
      
      
    listen 8080;
    root         /usr/share/nginx/html;
    index test.html;
}

[root@nginx2 ~]# echo "<h1>This is web2</h1>"  > /usr/share/nginx/html/test.html

Start Nginx
```
nginx -t
nginx
```

Installation and configuration of Keepalived

Install Keepalived
```
yum install -y keepalived
```

Configure Keepalived (some options are different for different roles: state and priority configuration items)

web1:master

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL_1
}
vrrp_script nginx_check {
  script "/tools/nginx_check.sh"
  interval 1
}
vrrp_instance VI_1 {
  state MASTER
  #nopreempt
  interface ens33
  virtual_router_id 52
  priority 100
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass test
  }
  virtual_ipaddress {
    192.168.149.100
  }
  track_script {
    nginx_check
  }
  notify_master /tools/master.sh
  notify_backup /tools/backup.sh
  notify_fault /tools/fault.sh
  notify_stop /tools/stop.sh
}

web2:slave

vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id LVS_DEVEL_2
}
vrrp_script nginx_check {
  script "/tools/nginx_check.sh"
  interval 1
}
vrrp_instance VI_1 {
  state BACKUP
  #nopreempt
  interface ens33
  virtual_router_id 52
  priority 90
  advert_int 1
  authentication {
    auth_type PASS
    auth_pass test
  }
  virtual_ipaddress {
    192.168.149.100
  }
  track_script {
    nginx_check
  }
  notify_master /tools/master.sh
  notify_backup /tools/backup.sh
  notify_fault /tools/fault.sh
  notify_stop /tools/stop.sh
}

Write relevant scripts

Write Keepalived log script (output each work process to the log)

# 将所有脚本放在同一个目录下，方便集中管理
mkdir /tools && cd /tools

cat > master.sh << 'EOF'
ip=$(hostname -I | awk '{print $1}')
dt=$(date +'%Y%m%d %H:%M:%S')
echo "$0--${ip}--${dt}" >> /tmp/kp.log
EOF

cat > backup.sh << 'EOF'
ip=$(hostname -I | awk '{print $1}')
dt=$(date +'%Y%m%d %H:%M:%S')
echo "$0--${ip}--${dt}" >> /tmp/kp.log
EOF

cat > fault.sh << 'EOF'
ip=$(ip addr|grep inet| grep 192.168 |awk '{print $2}')
dt=$(date +'%Y%m%d %H:%M:%S')
echo "$0--${ip}--${dt}" >> /tmp/kp.log
EOF

cat > stop.sh << 'EOF'
ip=$(ip addr|grep inet| grep 192.168| awk '{print $2}')
dt=$(date +'%Y%m%d %H:%M:%S')
echo "$0--${ip}--${dt}" >> /tmp/kp.log
EOF

Write health check scripts

This script detects the Nginx process. If it is found that the Nginx process is gone, it will return failure (1)

cat > nginx_check.sh << 'EOF'
#!/bin/bash
result=`pidof nginx`
if [ ! -z "${result}" ];
  then exit 0
  else exit 1
fi
EOF

After writing the scripts, add x permissions to these scripts and start Keepalived
```
cd /tools/ && chmod +x *.sh
systemctl restart keepalived.service
```

verify

Visit VIP address

Now web1 (192.168.149.130) is the master

Access 192.168.149.100:8080 and you can see that the access is successful.
Switch between active and standby

Close web1 service
```
[root@nginx1 ~]# nginx -s stop
```
Then the host of web2 service can see that web2 (192.168.149.131) has obtained VIP and upgraded to the master

Access the VIP address (192.168.149.100:8080) and you can find that you have accessed the web2 server

Then restart web1 and take a look

You can see that the web1 host has become the master again. Because there is no preemption mode/non-preemption mode configured, the election mechanism is priority. The priority of web1 is higher than that of web2, so after web1 starts, it will take over the master role.

problem solved

Check the keepalived system log: less /var/log/messages

Unable to access script /shell/nginx_check.sh

This script is disabled by selinux. The security context of the keepalived process is inconsistent with the security context of the script, and rwxit cannot be executed even if it has permission.
- The simplest solution is to turn off the selinux function:
```
# 永久关闭 selinux
sed -i "s\SELINUX=enforcing\SELINUX=disabled\g" /etc/selinux/config
# 重启主机
reboot

# 临时关闭selinux，reboot服务器后失效
etenforce 0
```
- Solution 2 : Move the script into /usr/libexec/keepalivedthe directory
  
  The security context of this directory is keepalived_unconfined_script_exec_tconsistent with the security context of the keepalived process.
- Solution 3: Modify check.shthe security context of the keepalived process to the security context
```
chcon -t keepalived_unconfined_sript_exec_it /etc/keepalived/nginx_check.sh
```
  keepalived_unconfined_script_exec_tIs a context used in SELinux to identify Keepalived to execute unrestricted scripts
  
  This context allows the Keepalived process to bypass some SELinux restrictions when executing scripts, allowing the script to be executed if needed
```
# 查看 keepalived 进程的安全上下文
ps -eZ | grep keepalived
#输出：system_u:system_r:keepalived_t:s0 19689 ?	00:00:00 keepalived

# 查看文件/目录的安全上下文
ll -Z /etc/keepalived/check.sh
#输出：-rwr-xr-x. root system_u:object_r:etc_t:s0
```
Disabling track script chk_nginx since not found

Add track_script { chk_nginx } to the VRRP instance to enable script detection
warning default user ‘keepalived_script’for script execution does not exist - please create

The user group keepalived_script cannot be found. Enable the following configuration. If it cannot be found, use root.

Add script_user root in global configuration global_defs

Note: This configuration is not necessary, and the log printing is only a warning.
SECURITY VIOLATION - scripts are being executed but script_security notenabled

Add enable_script_security in global configuration global_defs
Unknown keyword:’}’、‘track_script{’、'chk_nginxetc.

The configuration file format is incorrect: Where braces { are used, make sure there is a space between the text and the braces.
Can't open PID file /var/run/nginx.pid (yet?) after start: No such fileor directory

When using systemctl start nginx, nginx.service is called first, and the nginx.pid file is not generated during startup.

Add ExecStartPost=/bin/sleep 0.1 under the [service] module in /usr/lib/systemd/system/nginx.service
Can't open PID file /var/run/keepalived.pid (yet?) after start: No suchfile or directory

When using systemctl start keepalived, keepalived.service is called first, and the keealived.pid file is not generated during startup.

Add ExecStartPost=/bin/sleep 0.1 under the [service] module in /usr/lib/systemd/system/keepalived.service