Build keepalived+nginx hot standby high availability (active standby + dual active mode)
Preface
The master Nginx hangs up, but the slave nginx can work immediately
Use vrrp technology to provide VIP
When the main nginx hangs up, use a script to shut down the main keepalied. The keepalived uses vrrp technology to achieve the slave machine to obtain VIP, install the same nginx and configuration on the slave machine, and the slave machine continues to provide services to the outside world through VIP.
Related articles:
[Nginx-This article is enough to learn nginx]
[LVS+KeepAlived+Nginx high availability implementation plan]
A keepalived introduction
nginx:
It is a very excellent reverse proxy tool that supports request distribution, load balancing, caching and other very practical functions. In terms of request processing, nginx uses the epoll model, which is a model based on event monitoring, so it has very efficient request processing efficiency, and the single-machine concurrency capacity can reach millions. Requests received by nginx can be distributed to its lower-level application servers through load balancing policies. These servers are generally deployed in clusters. Therefore, in the case of insufficient performance, the application server can expand traffic by adding machines. At this time, for some very large websites, the performance bottleneck comes from nginx, because the concurrency capacity of a single machine nginx has an upper limit, and nginx itself does not support cluster mode, so the horizontal expansion of nginx at this time is appears particularly important.
Keepalived:
It is a highly reliable running software that implements VRRP backup routing under Linux. The service model designed based on Keepalived can truly achieve instant and seamless IP handover when the main server and backup server fail.
VRRP protocol:
the full name is Virtual Router Redundancy Protocol
, which is the virtual routing redundancy protocol. It can be considered a fault-tolerant protocol to achieve high availability of routers. N routers that provide the same functions are formed into a router group (RouterGroup). This group has a master and multiple backups, but it looks like one to the outside world. It constitutes a virtual router and has a virtual IP (vip, which is the default route for other machines in the LAN where the router is located). The master that owns this IP is actually responsible for ARP response and forwarding IP packets. Other routers in the group are on standby as backup roles. state. The master will send multicast messages. When the backup fails to receive vrrp packets within the timeout period, it is considered that the master is down. At this time, a backup needs to be elected as the master based on the VRRP priority to ensure the high availability of the router.
Summary: The two active and backup machines are keepalived to create a virtual IP, which is VIP, which does not mean VIP, but Virtual IP. The VIP starts to be owned by the main machine, and the backup machine is in an idle state. At the same time, the communication between the two keepalived is equivalent to a heartbeat line. They communicate with each other through the heartbeat line. As long as the main machine monitors (through a script) until the ngin service stops, the main machine Stop keepalived by yourself and hand the VIP to the backup machine to handle web requests until the main machine returns to normal again and return the VIP to the main machine.
The architecture is shown in the figure below:
Two ways of high availability:
1.Master-slave mode
This solution uses a VIP address and uses 2 machines at the front end. One is the master and one is the backup. However, only one machine is working at the same time. The other backup machine will always be wasted when the main machine does not malfunction. , for websites with few servers, this solution is not economical.
2.Dual master mode
This solution uses two VIP addresses, and uses 2 machines at the front end to serve as primary and backup for each other. Two machines work at the same time. When one of the machines fails, the requests from the two machines are transferred to one machine, which is very suitable. in the current architectural environment.
As shown below:
2. Environment construction
1. Environment preparation (centos7 system is used here)
| HOSTNAME | IP | illustrate |
|---|---|---|
| keep1 | 192.168.92.100 | keepalived master server (nginx same master load balancer) |
| keep2 | 192.168.92.101 | keepalived master server (nginx same master load balancer) |
| web1 | 192.168.92.102 | Back-end web server node (load balancing configured in nginx) This article is for convenience of testing (optional) |
| keep4 | 192.168.92.103 | Simulated client test machine (optional) |
For the convenience of testing, this article only uses two servers (keep1, keep12) for demonstration.
2.keepalived installation
It is recommended to use Xshell for simultaneous operation of multiple terminals and simultaneous operation of multiple windows.
This article will not go into details about the installation of nginx . If necessary, please refer to Baidu.
# 安装ipvs
yum install ipvsadm
# 安装keepalived
yum install keepalived
Common commands
#启动
systemctl start keepalived
#停止
systemctl stop keepalived
重启#
systemctl restart keepalived
#查看状态
systemctl status keepalived
#设置开机启动
systemctl enable keepalived
#关闭开机启动
systemctl disable keepalived
Relevant configurations can be modified by editing the keepalived.conf file in the /etc/keepalived/ directory.
vim /etc/keepalived/keepalived.conf
- CentOS 7 has the firewalld firewall installed by default. Turn off the firewall.
#启动防火墙
systemctl start firewalld
#关闭防火墙
systemctl stop firewalld
3. keepalived related configurations
1.nginx script and configuration
Since it will be tested later, the first simple script is enough. As long as it is judged that the nginx process has no value, the keepalived service will be stopped. The test script is as follows:
#! /bin/bash
pidof nginx
if [ $? -ne 0 ];then
systemctl stop keepalived
fi
If the test is completed, you can add an attempt to start nginx. If the attempt fails twice, stop the keepalived service.
Create the script check_nginx.sh. The script is as follows:
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
/usr/local/nginx/sbin/nginx
sleep 2
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
systemctl stop keepalived
fi
fi
nginx configuration in the nginx server: You only need to change the server_name to the IP of the VIP. No other changes are needed. During load balancing, you only need to access this VIP address to facilitate testing. This article directly returns the string through proxy. In actual operation,
location Configure the backend reverse proxy in .
server {
listen 80;
#server_name 192.168.92.200; #vip地址可加可不加
location / {
default_type text/html;
return 200 "Hello, Nginx! Server 192.168.92.100";
}
}
2. Active and backup mode
1.keepalived configuration
This configuration is in active-standby mode. After understanding the active-standby mode first, it will be easier to configure the dual-active mode.
Configuration file location:
/etc/keepalived/keepalived.conf
vi /etc/keepalived/keepalived.conf
There are three basic modules, global_defs global module, vrrp_instance configuration vip module, and vrrp_script script module, used to detect nginx services.
Note: After vrrp_script defines the script, the track_script parameter must be added to the vrrp_instance module. I fell into this trap, causing the script to not take effect.
global_defs module parameters
notification_email: keepalived needs to send an email notification address when a switch operation occurs, and the following smtp_server is also known to be the email server address. You can also call the police through other methods. After all, emails are not notified in real time.router_id: Machine identification, usually set to hostname. Email notifications are used when failures occur.
vrrp_instance module parameters
state: Specify the initial state of instance (Initial), MASTER or BACKUP, which is not unique and is related to the priority parameter behind it.interface: The network card bound to the instance, because when configuring the virtual IP, it must be added to the existing network card. (Pay attention to your own system, mine is ens33 by default, and some are eth0)mcast_src_ip: The source IP address when sending multicast packets. Pay attention here. This is actually sending VRRP notifications on that address. This is very important. You must choose a stable network card port to send. This is equivalent to the heartbeat port of heartbeat. If it is not set, the default IP of the bound network card will be used, which is the IP address specified by the interface.virtual_router_id: Set VRID here, it is very important here, the same VRID is a group, it will determine the MAC address of multicastpriority: Set the priority of this node, the one with higher priority is master (1-255)advert_int: Check interval, default is 1 second. This is the VRRP timer. Every such time interval, MASTER will send an advertisement message to notify other routers in the group that it is working normally.authentication: Define the authentication method and password. The master and slave must be the same.virtual_ipaddress: What is set here is the VIP, which is the virtual IP address. It is added and deleted as the state changes. It is added when the state is master and deleted when the state is backup. It is mainly determined by priority. It has little to do with the value set by state. Multiple IP addresses can be set here.track_script: Reference to the VRRP script, the name specified in the vrrp_script section. Run them periodically to change priorities and eventually trigger an active/standby switchover.
The vrrp_script module parameter
tells keepalived to switch under what circumstances, so it is particularly important. There can be multiple vrrp_scripts
script: A detection script written by myself. It can also be a one-line command such as killall -0 nginxinterval 2: Detected every 2sweight -5: If the detection fails (the script returns non-0), the priority is -5fall 2: The test must fail 2 times in a row to be considered a true failure. Will use weight to reduce the priority (between 1-255)rise 1: It is considered successful if the detection is successful once. but does not change the priority
On the main nginx server 192.168.92.100, the VIP is set to 192.168.92.200 and the configuration is as follows:
! Configuration File for keepalived
global_defs {
router_id Nginx_01
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.200
}
track_script {
check_nginx
}
}
On the backup nginx server 192.168.92.101, the configuration is the same, but there are three differences, one of which must be the same: 1. The router_id is different, 2. the state BACKUP is different, and 3. the priority is different. 4.virtual_router_id must be the same. The configuration is as follows:
! Configuration File for keepalived
global_defs {
router_id Nginx_02
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.200
}
track_script {
check_nginx
}
}
2. Start the test and check the IP
Restart two keepalived
syetemctl restart keepalived
ip a
Check the ip, you can see that vip is now on server 100
- Test whether VIP is available
[root@bogon keepalived]# curl http://192.168.92.200
Hello, Nginx! Server 192.168.92.100
- Stop a keepalived and check whether the vip has drifted
When keepalived has been stopped in the 100 service, request the service address again and check the results. The 101 server is requested.
[root@bogon keepalived]# curl http://192.168.92.200
Hello, Nginx! Server 192.168.92.101
3.Dual master mode
1.keepalived configuration
After understanding the active and standby modes, the dual-active mode is much easier to configure. Just add a vrrp_instance named vrrp_instance VI_2 to each keepalived configuration file, change a few parameters, and set another VIP: 192.168.200.201
keep1 configuration is as follows:
! Configuration File for keepalived
global_defs {
router_id Nginx_01
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface ens33
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.200
}
track_script {
check_nginx
}
}
# 100------100
vrrp_instance VI_2 {
state BACKUP
interface ens33
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.201
}
track_script {
check_nginx
}
}
The keep2 configuration is as follows:
! Configuration File for keepalived
global_defs {
router_id Nginx_02
}
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.200
}
track_script {
check_nginx
}
}
# 101------101
vrrp_instance VI_2 {
state MASTER
interface ens33
virtual_router_id 52
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.92.201
}
track_script {
check_nginx
}
}
After the modification is completed, start keep1 and keep2 respectively to check the binding VIP status.
2. Start the test and check the IP
ip addr
If you see the following results, the configuration is successful.
1. Test whether VIP is available
The following results indicate that the configuration is successful, as follows:
[root@bogon ~]# curl http://192.168.92.201
Hello, Nginx! Server 192.168.92.101
[root@bogon ~]# curl http://192.168.92.200
Hello, Nginx! Server 192.168.92.100
2. Stop a keepalived and check whether the vip has drifted.
Next we stop the keep service of 192.168.90.100
systemctl stop keepalived
Check the IP address of the 101 server.
When keepalived in the 100 service has been stopped, request the service address again and check the results. The 101 server is requested.
[root@bogon ~]# curl http://192.168.92.201
Hello, Nginx! Server 192.168.92.101
[root@bogon ~]# curl http://192.168.92.200
Hello, Nginx! Server 192.168.92.101
Manually start the 100 server, and you will find that the 200 address has been automatically bound.
4. Detailed explanation of keepalived configuration file (reference)
#全局配置
global_defs {
# 邮件通知信息
notification_email {
# 定义收件人
[email protected]
}
# 定义发件人
notification_email_from [email protected]
# SMTP服务器地址
smtp_server 192.168.200.1
smtp_connect_timeout 30
# 路由器标识，一般不用改，也可以写成每个主机自己的主机名
router_id LVS_DEVEL
# VRRP的ipv4和ipv6的广播地址，配置了VIP的网卡向这个地址广播来宣告自己的配置信息，下面是默认值
vrrp_mcast_group4 224.0.0.18
vrrp_mcast_group6 ff02::12
}
# 定义用于实例执行的脚本内容，比如可以在线降低优先级，用于强制切换
vrrp_script SCRIPT_NAME {
}
# 一个vrrp_instance就是定义一个虚拟路由器的，实例名称
vrrp_instance VI_1 {
# 定义初始状态，可以是MASTER或者BACKUP
state MASTER
# 工作接口，通告选举使用哪个接口进行
interface ens33
# 虚拟路由ID，如果是一组虚拟路由就定义一个ID，如果是多组就要定义多个，而且这个虚拟
# ID还是虚拟MAC最后一段地址的信息，取值范围0-255
virtual_router_id 51
# 使用哪个虚拟MAC地址
use_vmac XX:XX:XX:XX:XX
# 监控本机上的哪个网卡，网卡一旦故障则需要把VIP转移出去
track_interface {
eth0
ens33
}
# 如果你上面定义了MASTER,这里的优先级就需要定义的比其他的高
priority 100
# 通告频率，单位为秒
advert_int 1
# 通信认证机制，这里是明文认证还有一种是加密认证
authentication {
auth_type PASS
auth_pass 1111
}
# 设置虚拟VIP地址，一般就设置一个，在LVS中这个就是为LVS主机设置VIP的，这样你就不用自己手动设置了
virtual_ipaddress {
# IP/掩码 dev 配置在哪个网卡
192.168.200.16/24 dev eth1
# IP/掩码 dev 配置在哪个网卡的哪个别名上
192.168.200.17/24 dev label eth1:1
}
# 虚拟路由，在需要的情况下可以设置lvs主机 数据包在哪个网卡进来从哪个网卡出去
virtual_routes {
192.168.110.0/24 dev eth2
}
# 工作模式，nopreempt表示工作在非抢占模式，默认是抢占模式 preempt
nopreempt|preempt
# 如果是抢占默认则可以设置等多久再抢占，默认5分钟
preempt delay 300
# 追踪脚本，通常用于去执行上面的vrrp_script定义的脚本内容
track_script {
}
# 三个指令，如果主机状态变成Master|Backup|Fault之后会去执行的通知脚本，脚本要自己写
notify_master ""
notify_backup ""
notify_fault ""
}
# 定义LVS集群服务，可以是IP+PORT；也可以是fwmark 数字，也就是防火墙规则
# 所以通过这里就可以看出来keepalive天生就是为ipvs而设计的
virtual_server 10.10.10.2 1358 {
delay_loop 6
# 算法
lb_algo rr|wrr|lc|wlc|lblc|sh|dh
# LVS的模式
lb_kind NAT|DR|TUN
# 子网掩码，这个掩码是VIP的掩码
nat_mask 255.255.255.0
# 持久连接超时时间
persistence_timeout 50
# 定义协议
protocol TCP
# 如果后端应用服务器都不可用，就会定向到那个服务器上
sorry_server 192.168.200.200 1358
# 后端应用服务器 IP PORT
real_server 192.168.200.2 1358 {
# 权重
weight 1
# MSIC_CHECK|SMTP_CHEKC|TCP_CHECK|SSL_GET|HTTP_GET这些都是
# 针对应用服务器做健康检查的方法
MISC_CHECK {}
# 用于检查SMTP服务器的
SMTP_CHEKC {}
# 如果应用服务器不是WEB服务器，就用TCP_CHECK检查
TCP_CHECK {
# 向哪一个端口检查，如果不指定默认使用上面定义的端口
connect_port <PORT>
# 向哪一个IP检测，如果不指定默认使用上面定义的IP地址
bindto <IP>
# 连接超时时间
connect_timeout 3
}
# 如果对方是HTTPS服务器就用SSL_GET方法去检查，里面配置的内容和HTTP_GET一样
SSL_GET {}
# 应用服务器UP或者DOWN，就执行那个脚本
notify_up "这里写的是路径，如果脚本后有参数，整体路径+参数引起来"
notify_down "/PATH/SCRIPTS.sh 参数"
# 使用HTTP_GET方法去检查
HTTP_GET {
# 检测URL
url {
# 具体检测哪一个URL
path /testurl/test.jsp
# 检测内容的哈希值
digest 640205b7b0fc66c1ea91c463fac6334d
# 除了检测哈希值还可以检测状态码，比如HTTP的200 表示正常，两种方法二选一即可
status_code 200
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
url {
path /testurl3/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334d
}
# 向哪一个端口检查，如果不指定默认使用上面定义的端口
connect_port <PORT>
# 向哪一个IP检测，如果不指定默认使用上面定义的IP地址
bindto <IP>
# 连接超时时间
connect_timeout 3
# 尝试次数
nb_get_retry 3
# 每次尝试之间间隔几秒
delay_before_retry 3
}
}
real_server 192.168.200.3 1358 {
weight 1
HTTP_GET {
url {
path /testurl/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
url {
path /testurl2/test.jsp
digest 640205b7b0fc66c1ea91c463fac6334c
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 3
}
}
}
Summarize
Access to the two VIPs configured in keepalived can be scheduled normally. When we stop any keepalived node, access is still normal. At this point, the construction of keepalived+nginx hot standby high availability (active standby + dual active mode) is completed.