Linux operation and maintenance --- Keepalived + Nginx to achieve high-availability Web load balancing

Keepalived is a high-performance server high-availability or hot standby solution. Keepalived can be used to prevent a single point of failure of the server. By cooperating with Nginx, it can achieve high availability of web front-end services.

CPU name	IP address	Virtual IP	Character
lb01	192.168.203.150	192.168.203.88	keepalived（Master）
lb02	192.168.203.148	192.168.203.89	keepalived（Backup）
web1	192.168.203.136		nginx
web2	192.168.203.135		nginx

Dependent packages and installation packages that need to be prepared

[root@web1 ~]# ls | grep tar*
keepalived-2.0.2.tar.gz
libevent-2.1.12-stable.tar.gz
nginx-1.6.0.tar.gz
openssl-1.0.1h.tar.gz
pcre-8.35.tar.gz
zlib-1.2.8.tar.gz

Install openssl

[root@web1 ~]# tar -xf openssl-1.0.1h.tar.gz 
[root@web1 ~]# cd openssl-1.0.1h/
[root@web1 openssl-1.0.1h]# ./config --prefix=/usr/local/openssl
[root@web1 openssl-1.0.1h]# make && make install

Install pcre

[root@web1 ~]# tar -xf pcre-8.35.tar.gz 
[root@web1 ~]# cd pcre-8.35/
[root@web1 pcre-8.35]# ./configure --prefix=/usr/local/pcre
[root@web1 pcre-8.35]# make && make install

Install zlib

[root@web1 ~]# tar -xf zlib-1.2.8.tar.gz 
[root@web1 ~]# cd zlib-1.2.8/
[root@web1 zlib-1.2.8]# ./configure --prefix=/usr/local/zlib
[root@web1 zlib-1.2.8]# make && make install

Install Nginx

[root@web1 src]# wget http://nginx.org/download/nginx-1.6.0.tar.gz
[root@web1 src]# tar -xf nginx-1.6.0.tar.gz
[root@web1 src]# cd nginx-1.6.0/ 
[root@web1 nginx-1.6.0]# ./configure --prefix=/usr/local/nginx --with-openssl=../openssl-1.0.1h --with-pcre=../pcre-8.35 --with-zlib=../zlib-1.2.8 --with-http_ssl_module
[root@web1 nginx-1.6.0]# make && make install
[root@web1 nginx-1.6.0]# useradd www -s /sbin/nologin -M   # 创建用户

Modify Nginx configuration file

[root@web1 conf]# egrep -v "#|^$" nginx.conf
worker_processes  1;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
 server {
        listen       80;
        server_name  192.168.203.136;
        charset utf-8;
        location / {
            root   html;
            index  index.html index.htm;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}
[root@web1 conf]#

Start Nginx

[root@web1 ~]# /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
[root@web1 ~]# /usr/local/nginx/sbin/nginx -s reload

View the start of nginx process

[root@web1 ~]# ps -ef | grep nginx
root      54235      1  0 17:51 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
www       54244  54235  0 17:51 ?        00:00:00 nginx: worker process
root      54258   8945  0 17:51 pts/0    00:00:00 grep --color=auto nginx
[root@web1 ~]# 

Configure nginx virtual host based on domain name

[root@web1 nginx]# cd html
[root@web1 html]# mkdir www
[root@web1 html]# vim www/index.html
192.168.203.136 www.chd.comm
[root@web1 html]# cd ..
[root@web1 nginx]# sbin/nginx -t   # 检查语句
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@web1 nginx]# sbin/nginx    # 启动nginx
[root@web1 nginx]# sbin/nginx -s reload   # 重启nginx
[root@web1 nginx]# ps -ef |grep nginx   # 查看nginx进程号
nobody      509 130219  0 15:19 ?        00:00:00 nginx: worker process
root        512 111842  0 15:19 pts/2    00:00:00 grep --color=auto nginx
root     130219      1  0 14:34 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
[root@web1 nginx]# echo "192.168.203.136 www.chd.com mail.chd.com mail.cnd.com" >>/etc/hosts   将解析追加到hosts文件中
[root@web1 nginx]# tail -1 /etc/hosts    # 查看hosts文件最后一行内容
192.168.203.136  www.chd.com ftp.chd.com mail.chd.com
[root@web1 nginx]#

Open the browser and enter the IP address to access Nginx

Nginx starts automatically at boot

[root@web1 ~]# vim /etc/rc.local
/usr/local/nginx/sbin/nginx

Configure a simple load balancing

Modify the nginx.conf main configuration file of the lb01 server

[root@web1 nginx]# cat conf/nginx.conf | grep -v "^#"
user  www www;
worker_processes  1;
​
events {
    worker_connections  1024;
}
​
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    gzip  on;
    
    upstream webServer {
    server 192.168.203.135:80  weight=5;
    server 192.168.203.136:80  weight=6;
    }
    
    server {
        listen       80;
        server_name  www.chd.cn;
        charset utf-8;
        #access_log  logs/host.access.log  main;
        location / {
        proxy_pass    http://webServer;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Modify the nginx.conf main configuration file of the lb02 server

[root@web2 nginx]# cat conf/nginx.conf | grep -v "^#"
user  www www;
worker_processes  1;
​
events {
    worker_connections  1024;
}
​
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    gzip  on;
​
    upstream webServer {
    server 192.168.203.135:80  weight=5;
    server 192.168.203.136:80  weight=6;
    }
    
    server {
        listen       80;
        server_name  192.168.203.148;
        charset utf-8;
        #access_log  logs/host.access.log  main;
        location / {
            root   html;
            index  index.html index.htm;
        proxy_pass    http://webServer;
        }
​
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

Verification: When parsing www.chd.cn server, it will switch between web1 and web2

​[root@lb01 nginx]# curl www.chd.cn
192.168.203.135 www.chd.com
[root@lb01 nginx]# curl www.chd.cn
192.168.203.136 www.chd.com

Configure keepalived load balancing

Install Keepalived

1.yum安装：
yum install keepalived -y     # 推荐使用yum安装，简单、快捷、高效


2.源码包安装：
[root@lb01 ~]# ls | grep keepalived*
keepalived-2.0.2.tar.gz
[root@lb01 ~]# tar -xf keepalived-2.0.2.tar.gz 
[root@lb01 ~]# cd keepalived-2.0.2/
[root@lb01 keepalived-2.0.2]# ./configure --prefix=/usr/local/keepalived 
[root@lb01 keepalived-2.0.2]# make && make install
[root@lb01 keepalived-2.0.2]# cp keepalived/etc/init.d/keepalived /etc/init.d/
[root@lb01 keepalived-2.0.2]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[root@lb01 keepalived-2.0.2]# mkdir /etc/keepalived
[root@lb01 keepalived-2.0.2]# chmod +755 /etc/keepalived/
[root@lb01 keepalived-2.0.2]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[root@lb01 keepalived-2.0.2]# ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/
[root@lb01 keepalived-2.0.2]# ./keepalived/etc/init.d/keepalived start
Starting keepalived:                                       [  OK  ]
[root@lb01 keepalived-2.0.2]#

注意：在编译时发出警告解决办法
*** WARNING - this build will not support IPVS with IPv6. Please install libnl/libnl-3 dev libraries to support IPv6 with IPVS.
​​安装
[root@localhost keepalived-2.0.2]# yum -y install libnl libnl-devel
[root@localhost keepalived-2.0.2]# yum install -y libnfnetlink-devel

Main configuration file	/etc/keepalived/keepalived.conf
Environment configuration file	/etc/sysconfig/keepalived

Modify the /etc/keepalived/keepalived.conf configuration file of lb1

[root@lb01 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
​
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb1
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
​
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.203.88/24
    }
 }
 virtual_server 192.168.203.88 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR
    persistence_timeout 50
    protocol TCP
    }   
[root@lb01 nginx]# /etc/init.d/keepalived restart      
Restarting keepalived (via systemctl):                     [  OK  ]
[root@lb01 nginx]# ip addr | grep 192.168.203.88
    inet 192.168.203.88/24 scope global secondary ens33
[root@lb01 nginx]#

Modify the /etc/keepalived/keepalived.conf configuration file of lb2

[root@lb02 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
​
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb02
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
​
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 52
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.203.88/24
    }
 }   
    virtual_server 192.168.203.88 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR 
    persistence_timeout 50
    protocol TCP
    }  
[root@lb02 ~]# /etc/init.d/keepalived restart
Restarting keepalived (via systemctl):                     [  OK  ]
[root@lb02 ~]# ip addr | grep 192.168.203.88
[root@lb02 ~]#

You can see that it is now the virtual IP address taken over by the master node. If the master node is down, see if the backup node will automatically take over keepalived

[root@lb01 nginx]# /etc/init.d/keepalived stop
Stopping keepalived (via systemctl):                       [  OK  ]
[root@lb01 nginx]# 
​
[root@lb02 nginx]# ip addr | grep 192.168.203.88
    inet 192.168.203.88/24 scope global secondary ens33
[root@lb02 nginx]#

You can see that the backup node has automatically taken over the virtual IP address!

[root@lb01 nginx]# /etc/init.d/keepalived restart      
Restarting keepalived (via systemctl):                     [  OK  ]
[root@lb01 nginx]# ip addr | grep 192.168.203.88       
    inet 192.168.203.88/24 scope global secondary ens33
[root@lb01 nginx]#

If the master node is started, the master node will automatically take over the keepalived service

Configure keepalived dual main mode

Modify the /etc/keepalived/keepalived.conf configuration file of lb1

[root@lb01 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
​
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb1
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
​
vrrp_instance VI_1 {
    state MASTER
    interface ens33
    virtual_router_id 52
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.203.88/24
    }
 }
​
vrrp_instance VI_2 {
    state BACKUP
    interface ens33
    virtual_router_id 53
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        192.168.203.89/24
    }   
}   

Modify the /etc/keepalived/keepalived.conf configuration file of lb2

[root@lb02 nginx]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
​
global_defs {
   notification_email {
     [email protected]
   }
   notification_email_from [email protected]
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id lb02
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}
​
vrrp_instance VI_1 {
    state BACKUP
    interface ens33
    virtual_router_id 52
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.203.88/24
    }
 }   
    virtual_server 192.168.203.88 80 {
    delay_loop 6
    lb_algo rr 
    lb_kind DR 
    persistence_timeout 50
    protocol TCP
    }  
​
vrrp_instance VI_2 {
    state MASTER
    interface ens33
    virtual_router_id 53
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }   
    virtual_ipaddress {
        192.168.203.89/24
    }   
} 

Restart keepalived after modifying the configuration file

[root@lb01 nginx]# /etc/init.d/keepalived restart              
Restarting keepalived (via systemctl):                     [  OK  ]
[root@lb01 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.88/24 scope global secondary ens33

After restarting the backup node, you can see that the virtual IP address 192.168.203.89 has been taken over by the backup node

[root@lb02 nginx]# /etc/init.d/keepalived restart              
Restarting keepalived (via systemctl):                     [  OK  ]
[root@lb02 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.89/24 scope global secondary ens33
[root@lb02 nginx]#

Down the master node, check the status of the backup node

[root@lb01 nginx]# /etc/init.d/keepalived stop
Stopping keepalived (via systemctl):                       [  OK  ]
[root@lb01 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
[root@lb01 nginx]#

You can see that after the master node is down, the backup node has taken over two virtual IP addresses

[root@lb02 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.89/24 scope global secondary ens33
    inet 192.168.203.88/24 scope global secondary ens33

If the master node is started again at this time, it will take over 192.168.203.88 again

[root@lb01 nginx]# /etc/init.d/keepalived start
Starting keepalived (via systemctl):                       [  OK  ]
[root@lb01 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.88/24 scope global secondary ens33
[root@lb01 nginx]#

The status of the standby node will also be released soon 192.168.203.88

[root@lb02 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.89/24 scope global secondary ens33
[root@lb02 nginx]#

After the backup node is down, check the status of the master node

[root@lb02 nginx]# /etc/init.d/keepalived stop
Stopping keepalived (via systemctl):                       [  OK  ]
[root@lb02 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
[root@lb02 nginx]#

You can see that the master node quickly took over 192.168.203.89

[root@lb01 nginx]# ip add|egrep "192.168.203.88|192.168.203.89"
    inet 192.168.203.88/24 scope global secondary ens33
    inet 192.168.203.89/24 scope global secondary ens33
[root@lb01 nginx]#

So far, nginx + keepalived high-availability cluster realizes load balancing.

Summary: Regardless of whether the lb1 primary node or the lb2 standby node is down, the virtual IP address will be switched and taken over. When the down node is started again, the virtual IP will be taken over again!