大型网站分布式架构（十一）—— Keepalived+Nginx+Tomcat集群实现服务的高可用HA

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/weixin_37490221/article/details/82178762

服务架构

两台Keepalived做主从热备，每个Keepalived节点都集成一台Nginx实现Nginx的HA，Nginx再反向代理Tomcat集群，同时实现Tomcat集群的负载均衡和动静分离。

服务分布

• 两个Keepalived主备节点
  
  • 192.168.214.150
  • 192.168.214.151
• 两个Nginx服务节点
  
  • 192.168.214.150:80
  • 192.168.214.151:80
• 三台Tomcat服务器
  
  • 192.168.214.150:8080
  • 192.168.214.151:8080
  • 192.168.214.152:8080

概况

Keepalived: Load balancer and high availability service。这是官方对Keepalived的最直接的解释，Keepalived是一个负载均衡和高可用服务。我们在这里使用它的高可用特性，集成Nginx实现Nginx的高可用

前提

搭建Nginx+Tomcat集群架构：一台Nginx+三台Tomcat —— 大型网站分布式架构（八）—— Tomcat集群横向拓展 + Nginx负载均衡

扩展Nginx服务

1.搭建第二台Nginx

之前我们已经在192.168.214.150节点上搭建了一台Nginx服务，现在需要在192.168.214.151节点上搭建另外一台，搭建方式相同：大型网站分布式架构（六）—— Nginx的安装

2.修改nginx.conf配置

两台Nginx的配置文件相同，都是设置了Tomcat集群的负载均衡，动静分离。所以直接分发192.168.214.150节点的nginx.conf给192.168.214.151节点即可

[root@centos6-1 ~]# cd /usr/local/nginx/conf/
[root@centos6-1 conf]# scp -r nginx.conf root@centos6-2:$PWD
nginx.conf                                                                                                 100%  913     0.9KB/s   00:00    
[root@centos6-1 conf]# 

搭建Keepalived主备

1.Keepalived安装

博主采用yum形式安装，省事简便，如果寻求版本的更新，可以使用源码编译安装。同时在两个节点上安装：

• 192.168.214.150 —— centos6-1
• 192.168.214.151 —— centos6-2

[root@centos6-1 ~]# yum install keepalived
[root@centos6-2 ~]# yum install keepalived

配置文件修改

配置文件所在位置：keepalived.conf

[root@centos6-1 ~]# cd /etc/keepalived/
[root@centos6-1 keepalived]# ll
total 4
-rw-r--r--. 1 root root 507 Aug 27 21:17 keepalived.conf
[root@centos6-1 keepalived]#

修改后192.168.214.150节点的keepalived.conf配置文件：

! Configuration File for keepalived

global_defs {
    #keepalived所在主机名
    router_id centos6-1
}

#检查nginx状态的执行脚本
vrrp_script kpld_ngx {
    #脚本所在路径
    script "/export/shell/kpld_ngx.sh"
    interval 2
    weight -15
}

vrrp_instance VI_1 {
    #设置成备份节点(有MASTER/BACKUP两个值)
    state BACKUP
    #主机网卡名
    interface eth0
    #虚拟路由id
    virtual_router_id 100
    #所在主机ip
    mcast_src_ip 192.168.214.150
    #优先级
    priority 100
    advert_int 1
    #认证
    authentication {
        auth_type PASS
        auth_pass zaomianbao
    }
    #触发脚本
    track_script {
        kpld_ngx
    }
    #虚拟ip
    virtual_ipaddress {
        192.168.214.200
    }
}

修改后192.168.214.151节点的keepalived.conf配置文件：

! Configuration File for keepalived

global_defs {
    router_id centos6-2
}

vrrp_script kpld_ngx {
    script "/export/shell/kpld_ngx.sh"
    interval 2
    weight -15
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 100
    #所有配置就这里跟192.168.214.150节点不同，设置一个与节点一不同的值即可
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass zaomianbao
    }
    track_script {
        kpld_ngx
    }
    virtual_ipaddress {
        192.168.214.200
    }
}

配置详解

配置项	解释
router_id	keepalived所在主机名
interval	执行间隔，单位秒
weight	每次故障启动后减去此权重：priority-weight
state	都设置成BACKUP即可，也可以指定一台设置成MASTER
interface	网卡名，这个需要通过ifconfig或ip addr命令获取
virtual_router_id	随意，只要保证两节点keepalived配置相同即可
priority	优先级，优先级决定了主备身份，当都设置成BACKUP时，启动时谁优先级高谁是主节点
virtual_ipaddress	虚拟ip，后面直接访问的ip

创建kpld_ngx.sh脚本

[root@centos6-1 ~]# cd /export/shell/
[root@centos6-1 shell]# touch kpld_ngx.sh
[root@centos6-1 shell]# chmod 777 kpld_ngx.sh

编写脚本内容

#!/bin/bash

COUNT=$(ps -C nginx --no-header |wc -l)

echo $COUNT

#判断Nginx 是否都挂掉了
if [ $COUNT -eq 0 ]
then
#如果挂掉了，就启动nginx
/usr/local/nginx/sbin/nginx
echo "重启nginx"

#等5秒钟后，再次查看是否 启动成功
sleep 5

#如果nginx没有启动起来，就直接干掉keepalived
COUNT=$(ps -C nginx --no-header |wc -l)
if [ $COUNT -eq 0 ]
then
echo "干掉keepalived"
killall keepalived
fi
fi

分发脚本给节点192.168.214.150

[root@centos6-1 shell]# scp kpld_ngx.sh root@centos6-2$PWD

启动各节点Tomcat

前提将项目appdemo部署到Tomcat上

[root@centos6-1 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-1 bin]# ./startup.sh 
[root@centos6-2 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-2 bin]# ./startup.sh 
[root@centos6-3 ~]# cd /export/servers/apache-tomcat-8.0.53/bin/
[root@centos6-3 bin]# ./startup.sh 

访问Tomcat

192.168.214.150节点Tomcat

192.168.214.151节点Tomcat

192.168.214.152节点Tomcat

启动Nginx

两个节点的Nginx都启动

[root@centos6-1 ~]# cd /usr/local/nginx/sbin/
[root@centos6-1 sbin]# ./nginx
[root@centos6-2 ~]# cd /usr/local/nginx/sbin/
[root@centos6-2 sbin]# ./nginx

访问Nginx

访问192.168.214.150节点Nginx

访问192.168.214.151节点Nginx

节点二获取不到静态资源是因为，在节点二上没有部署静态资源，而节点一我们之前部署过

启动Keepalived

两个节点都启动

[root@centos6-1 ~]# service keepalived start
[root@centos6-2 ~]# service keepalived start

访问Keepalived虚拟ip

虚拟ip我们配置为192.168.214.200

发现访问成功

查看Keepalived的状态

查看两个节点Keepalived的日志
192.168.214.150节点Keepalived日志

Aug 29 16:34:32 centos6-1 Keepalived[2028]: Starting Keepalived v1.2.13 (03/19,2015)
Aug 29 16:34:32 centos6-1 Keepalived[2029]: Starting Healthcheck child process, pid=2030
Aug 29 16:34:32 centos6-1 Keepalived[2029]: Starting VRRP child process, pid=2031
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Netlink reflector reports IP 192.168.214.150 added
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Netlink reflector reports IP fe80::20c:29ff:fedf:1580 added
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering Kernel netlink reflector
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering Kernel netlink command channel
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Registering gratuitous ARP shared channel
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Truncating auth_pass to 8 characters
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Configuration is using : 63985 Bytes
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: Using LinkWatch kernel netlink reflector...
#可以看到节点1为备份节点
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Netlink reflector reports IP 192.168.214.150 added
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Netlink reflector reports IP fe80::20c:29ff:fedf:1580 added
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Registering Kernel netlink reflector
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Registering Kernel netlink command channel
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Configuration is using : 6289 Bytes
Aug 29 16:34:32 centos6-1 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 29 16:34:32 centos6-1 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 29 16:34:32 centos6-1 kernel: IPVS: ipvs loaded.
Aug 29 16:34:32 centos6-1 Keepalived_vrrp[2031]: VRRP_Script(check_nginx) succeeded
Aug 29 16:34:32 centos6-1 Keepalived_healthcheckers[2030]: Using LinkWatch kernel netlink reflector...
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Received higher prio advert
Aug 29 16:34:36 centos6-1 Keepalived_vrrp[2031]: VRRP_Instance(VI_1) Entering BACKUP STATE

192.168.214.151节点Keepalived日志

Aug 29 16:34:34 centos6-2 Keepalived[2381]: Starting Keepalived v1.2.13 (03/19,2015)
Aug 29 16:34:34 centos6-2 Keepalived[2382]: Starting Healthcheck child process, pid=2383
Aug 29 16:34:34 centos6-2 Keepalived[2382]: Starting VRRP child process, pid=2384
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Netlink reflector reports IP 192.168.214.151 added
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Netlink reflector reports IP fe80::250:56ff:fe31:7ab0 added
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering Kernel netlink reflector
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering Kernel netlink command channel
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Registering gratuitous ARP shared channel
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Truncating auth_pass to 8 characters
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Configuration is using : 63945 Bytes
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: Using LinkWatch kernel netlink reflector...
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Entering BACKUP STATE
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Aug 29 16:34:34 centos6-2 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Aug 29 16:34:34 centos6-2 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Aug 29 16:34:34 centos6-2 kernel: IPVS: ipvs loaded.
Aug 29 16:34:34 centos6-2 Keepalived_vrrp[2384]: VRRP_Script(check_nginx) succeeded
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP 192.168.214.151 added
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP fe80::250:56ff:fe31:7ab0 added
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Registering Kernel netlink reflector
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Registering Kernel netlink command channel
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Opening file '/etc/keepalived/keepalived.conf'.
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Configuration is using : 6249 Bytes
Aug 29 16:34:34 centos6-2 Keepalived_healthcheckers[2383]: Using LinkWatch kernel netlink reflector...
#可以看到节点2为主节点
Aug 29 16:34:36 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) forcing a new MASTER election
Aug 29 16:34:36 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) forcing a new MASTER election
Aug 29 16:34:37 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Transition to MASTER STATE
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Entering MASTER STATE
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) setting protocol VIPs.
Aug 29 16:34:38 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.200
Aug 29 16:34:38 centos6-2 Keepalived_healthcheckers[2383]: Netlink reflector reports IP 192.168.214.200 added
Aug 29 16:34:43 centos6-2 Keepalived_vrrp[2384]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.214.200

keepalived内部运行原理

• 我们可以随意停止任意节点的Keepalived服务，发现服务仍然可以使用，此时如果停止的节点是主节点，那么存活的备份节点会切换成为主节点提供服务，如果再启动停掉的节点，那么该节点会自动成为备份节点。
• 我们也可以停掉任意节点的Nginx服务，当Nginx服务停掉之后，我们的检测Nginx状态的脚本会主动启动停掉的Nginx。

[root@centos6-2 ~]# ps -ef|grep nginx
root       6790      1  0 17:00 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody     6792   6790  0 17:00 ?        00:00:00 nginx: worker process      
root       6899   2262  0 17:01 pts/0    00:00:00 grep nginx
[root@centos6-2 ~]# /usr/local/nginx/sbin/nginx -s stop
[root@centos6-2 ~]# ps -ef|grep nginx                  
root       6922   6921  0 17:01 ?        00:00:00 /bin/bash /export/shell/kpld_ngx.sh
root       6927      1  0 17:01 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nobody     6929   6927  0 17:01 ?        00:00:00 nginx: worker process      
root       6931   2262  0 17:01 pts/0    00:00:00 grep nginx
[root@centos6-2 ~]# 

可以看到kpld_ngx.sh脚本在Nginx停掉后主动运行，又重新启动了，因为我们发现Nginx进程号发生了变化。

总结

到此我们就真正实现了网站服务的高可用。当然，我们还可以考虑一下优化的问题，比如访问到了节点2的Nginx服务时由于没有本地静态资源导致了无法获得对应的静态的资源，那么我们就可以把静态摘取出来搭建一个独立的静态资源系统。我们的项目并没有接入数据库，业务中也没有数据逻辑的处理，当计算达到一个顶峰时，当与数据库的连接达到一个极限时，很多很多的优化存在在项目开发的边边角角，而社区中都会有很多成熟的解决方案。博主将该专栏就先更到这个阶段，后期如果有机缘的话，博主会深入到微服务再到数据库等等方面。祝大家都能成为优秀的程序员。

---

参考：

• http://www.linuxde.net/2013/04/13381.html
• http://www.keepalived.org/manpage.html
• https://blog.csdn.net/yelllowcong/article/details/78764780
• https://www.cnblogs.com/youzhibing/p/7327342.html
• https://www.jianshu.com/p/b050d8861fc1
• https://baike.baidu.com/item/Keepalived/10346758?fr=aladdin