Docker install Keepalived + Nginx and achieve high availability

Docker install Keepalived + Nginx and achieve high availability

Demand goal: Install Keepalived + Nginx based on Centos image and achieve high availability

Ready to work

Download the centos image and configure the environment

$ docker pull centos # 拉取最新 centos
$ docker images # 查看镜像
$ docker run -itd --name centos centos # 后台启动一个容器
$ docker exec -it centos bash # 进入容器
$ yum -y upgrade # 升级所有包 
$ yum install -y yum-utils vim make wget gcc openssl openssl-devel popt-devel # 安装相关依赖

Download and install Nginx

Configure yum source

Official reference document http://nginx.org/en/linux_packages.html

$ cat /etc/yum.repos.d/nginx.repo

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/$releasever/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

$ yum install -y nginx # 安装 Nginx
$ whereis nginx # 查看安装路径,默认配置文件在 /etc/nginx
$ nginx -t # 插件配置
$ nginx # 启动
$ curl localhost # 检查启动是否成功

Download and install Keepalived

Keepalived official website download the latest version

$ mkdir -p /mnt/software && cd /mnt/software 
$ wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
$ docker cp keepalived-2.0.20.tar.gz centos:/mnt/ # 下载慢！！也可以从本地 copy
$ tar -zxvf keepalived-2.0.20.tar.gz 

Compile and install

$ cd keepalived-2.0.20
$ ./configure --prefix=/usr/local/keepalived

The following warning may appear after compilation

*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.

Follow the prompts

$ yum -y install libnl libnl-devel
No match for argument: libnl
No match for argument: libnl-devel
Error: Unable to find a match: libnl libnl-devel

What does not have a matching software source? We only need to use the VRRP function, as long as the Use VRRP Framework, Use VRRP VMAC, Use VRRP authentication is Yes, of course, you can also find RPM separately to install libnl.

https://centos.pkgs.org/7/centos-x86_64/libnl-1.1.4-3.el7.x86_64.rpm.html

$ wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libnl-1.1.4-3.el7.x86_64.rpm
$ wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libnl-devel-1.1.4-3.el7.x86_64.rpm
$ rpm -ivh libnl-1.1.4-3.el7.x86_64.rpm
$ rpm -ivh libnl-devel-1.1.4-3.el7.x86_64.rpm
$ ./configure --prefix=/usr/local/keepalived
$ make && make install

Configure Keepalived

$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/init.d/keepalived /etc/init.d/
$ cp /mnt/software/keepalived-2.0.20/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
$ cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
$ mkdir -p /etc/keepalived
$ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
$ chmod +x /etc/init.d/keepalived
$ ip address # 本机 ip 172.17.0.2 网卡 eth0
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
22: eth0@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Keepalived configuration files are placed in /etc/keepalived/keepalived.conf by default

! Configuration File for keepalived

global_defs {
	 # 负载均衡标识，在局域网内应该是唯一的
   router_id LVS_DEVEL
   enable_script_security
   # 建议新建用户来执行脚本
   script_user root
}

vrrp_script chk_nginx {
    script "/etc/keepalived/check.sh"
    # 每2秒检测一次nginx的运行状态
    interval 2 
    # 失败一次，将自己的优先级调整为-5
    weight -5
    # require 2 failures for KO
    fall 2
    # require 2 successes for OK
    rise 2
}

vrrp_instance VI_1 {
    # 标示状态为 MASTER 备份机为 BACKUP
    state MASTER
    # 设置实例绑定的网卡
    interface eth0
    # 同一实例下 virtual_router_id 必须相同
    virtual_router_id 51
    # MASTER 权重要高于 BACKUP 比如 BACKUP 为99
    priority 100
    # MASTER 与 BACKUP 负载均衡器之间同步检查的时间间隔，单位是秒
    advert_int 1 
    # 设置认证
    authentication { 
    		# 主从服务器验证方式
        auth_type PASS
        auth_pass 1111
    }
    # 虚拟IP，两个节点设置必须一致，可以设置多个
    virtual_ipaddress { 
        172.17.0.200
        172.17.0.210
    }
    track_script {
       chk_nginx
    }
}

$ cd /etc/keepalived
$ touch check.sh
$ chmod +x check.sh

The contents of the check.sh script are as follows

#!/bin/bash
Nginx=`ps -ef | grep nginx | grep -v grep | wc -l`
echo 'nginx status ' $Nginx
if [ $Nginx -eq 0 ]
then
    echo 'nginx shutdown...'
    nginx
    echo 'start nginx ...'
    sleep 2
    if [ `ps -ef | grep nginx | grep -v grep | wc -l` -eq 0 ]
    then
				ps -ef | grep keepalived | grep -v grep | awk '{print $2}' | xargs kill -9
				echo 'kill keepalived...'
    else
	echo 'nginx started...'
    fi
fi
echo 'script end...'

The role of the script: when the nginx process does not exist, it will automatically restart the nginx service, sleep for 2 seconds and check the nginx process again, if it does not exist, stop the keepalived service

Test script:

$ ps ax|grep nginx
   40 pts/1    S+     0:00 grep --color=auto nginx
$ ./check.sh
nginx status  0
nginx shutdown...
start nginx ...
nginx started...
script end...
$ ps ax|grep nginx
   48 ?        Ss     0:00 nginx: master process nginx
   49 ?        S      0:00 nginx: worker process
   57 pts/1    S+     0:00 grep --color=auto nginx

Check configuration

$ keepalived -t

At this point, Keepalived and Nginx are configured.

test

Container create snapshot

$ docker ps -a | grep centos
ace32d2fa597 centos "/bin/bash" About an hour ago Up About an hour  centos
$ docker commit ace32d2fa597 centos_keepalived_nginx:0.0.1 # 创建该容器的快照
$ docker images | grep centos_keepalived_nginx
centos_keepalived_nginx   0.0.1               801e75c18232        51 seconds ago      569MB

Create a master-slave container

Create a master-slave container based on the snapshot

$ docker run --privileged -itd -p 6698:80 --name keepalived_master --hostname keepalived_master centos_keepalived_nginx:0.0.1 /usr/sbin/init
$ docker run --privileged -itd -p 6699:80 --name  keepalived_slave --hostname keepalived_slave centos_keepalived_nginx:0.0.1 /usr/sbin/init

Enter the maser container to modify the configuration

$ docker exec -it keepalived_master bash # 进入 master 容器
$ vi /usr/share/nginx/html/index.html # 任意修改一点内容与 slave 做区分
$ nginx
$ curl localhoast

Enter the slave container to modify the configuration

$ docker exec -it keepalived_master bash # 进入 master 容器
$ vi /usr/share/nginx/html/index.html # 任意修改一点内容与 slave 做区分
$ nginx
$ curl localhoast

Test Keepalived

Start keepalived in the master container

$ docker exec -it keepalived_master bash
$ keepalived
$ curl 172.17.0.200 # curl 172.17.0.210
$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1000
    link/tunnel6 :: brd ::
36: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.17.0.200/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.17.0.210/32 scope global eth0
       valid_lft forever preferred_lft forever

Start keepalived in the slave container

$ docker exec -it keepalived_slave bash
$ vi /etc/keepalived/keepalived.conf # state 改为 BACKUP priority 改为 小于 100 的任意值 比如 99
$ keepalived
$ curl 172.17.0.200 # curl 172.17.0.210

to sum up:

• You will find that access to 172.17.0.200 and 172.17.0.210 both access content on the maser container

• If you turn off keepalived on the master, it will automatically switch to the content on the slave container

• When the nginx process does not exist, try to restart nginx. If nginx fails to start, shut down the keepalived service and implement automatic switching.

Remaining problem:

• How can the virtual IP set in Docker be accessed by the external network and how to map it? Docker research is not in-depth, please advise.
• The check.sh script checks that the Nginx service has time to return 0 and sometimes returns 2 (Nginx services are all closed), but using touch to create a new file is no problem! ! ! Why is this?

Welcome to communicate together ...