keepalived + lvs high availability

One, lvs introduction:

LVS is the abbreviation of Linux Virtual Server, which means Linux virtual server, which is a virtual server cluster system. Founded by Dr. Zhang Wensong in May 1998, this project is one of the earliest free software projects in China.

• Composition of LVS: Consists of one or more VIPs + multiple real servers
• The simple working principle of LVS: the user requests LVS VIP, and LVS forwards the request to the back-end server according to the forwarding method and algorithm, and the back-end server receives the request and returns it to the user. For users, they cannot see the specific application of the WEB backend.

2. Realize lvs load balancing method:

• There are three ways to achieve LVS load balancing forwarding, namely NAT, DR, and TUN modes. LVS balancing algorithms include: RR (round-robin), LC (least_connection), W (weight) RR, WLC mode, etc. (RR is polling mode , LC is the least connected mode).

(1) Principle of LVS NAT:

The user requests the LVS VIP to reach the director (LVS server: LB), and the director changes the destination IP address of the requested message to the realserver IP address of the backend, and at the same time changes the destination port of the message to the corresponding port of the realserver selected by the backend. Finally, the message is sent to the realserver, the realserver returns the data to the director, and the director sends the data to the user. (Two requests pass through the director, so if the access is large, the director will become a bottleneck), as shown in the figure:

(2) Principle of LVS DR:

The user requests LVS VIP to reach the director (LB balancer), and the director changes the destination MAC address of the requested message to the realserver MAC address of the backend, the destination IP is VIP (unchanged), and the source IP is the user IP address (unchanged) , and then the Director sends the message to the realserver, and the realserver detects that the target is its own local VIP. If it is in the same network segment, it returns the request directly to the user. If the user is not in the same network segment as the realserver, the user will be returned through the gateway, as shown in the figure:

3. Actual configuration of LVS load balancing

1. Use ipvsadm configuration to achieve, nat mode:

The implementation of LVS load balancing technology is based on the Linux kernel module IP_VS. Like iptables, it works directly in the kernel. The mainstream Linux distributions on the Internet have integrated the ipvs module by default, so you only need to install the management tool ipvsadm: yum install ipvsadm
-y

• 配置：
  ipvsadm -A -t 192.168.10.16:80 -s rr
  ipvsadm -a -t 192.168.10.16:80 -r 192.168.10.135 -m -w 2
  ipvsadm -a -t 192.168.10.16:80 -r 192.168.10.136 -m -w 2
• Relevant parameter description:
  -A adds a virtual server VIP address;
  -t virtual server provides tcp service;
  -s uses the scheduling algorithm;
  -a adds a backend real server to the virtual server;
  -r specifies the real server Address;
  -w the weight of the backend real server;
  -m set the current forwarding mode to NAT mode; -g is the direct routing mode; -i mode is the tunnel mode.

In this way, a virtual server cluster with VIP 192.168.10.16, port 80, real servers 192.168.10.134, 192.168.10.135, and port 80 is configured. When accessing 192.168.10.16:80, it will follow the rr polling rule Forward the request to port 80 of hosts 134 and 135.
Test:

But this method has a disadvantage. When the back-end server 134 or 135 service is down, for example, the nginx service of 136 is gone, an error will be reported when accessing the VIP: that is to say, ipvsadm will not automatically

remove the problematic server :

This introduces keepalived:

• It can automatically detect the running status of the server according to the configured rules, and perform actions of removing and adding, so that users can not feel whether the back-end server is down. When a back-end WEB server works normally, Keepalived will automatically save When the server is added to the server group, all these tasks are automatically completed without manual intervention. All that needs to be done manually is to repair the faulty WEB server.
• In addition, keepalived can be used for HA failover, that is, there is a backup LVS, the main LVS is down, and the LVS VIP is automatically switched to the slave. Based on LVS+Keepalived, load balancing and high availability functions can be realized to meet the stable and efficient operation of the website 7x24 hours .
  Keepalived is based on three-layer detection (IP layer, TCP transport layer, and application layer), and is mainly used to detect the status of the WEB server. If a WEB server crashes or fails to work, Keepalived will detect and send the faulty WEB server from removed from the system;

It should be noted that if the keepalived.conf configuration is used, there is no need to execute the ipvsadm -A command to add a balanced realserver command, and all configurations can be set in keepalived.conf.

2. Use keepalived configuration to realize NAT mode

• Need to build a server cluster with VIP 192.168.10.16 and rip 192.168.10.134, 192.168.10.135, 192.168.10.136

2.1 Environment

2.2 configuration of keepalived master

192.168.10.134
Clear the configuration of ipvsadm:

Modify the content of /etc/keepalived/keepalived.conf:

[root@node1 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   router_id node1
}
vrrp_script chk_nginx {
    
    
	script "/data/sh/check_nginx.sh"
	interval 2
	weight -20
}

vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.10.16
    }

}
virtual_server 192.168.10.16 80 {
    
    
    delay_loop 1
    lb_algo rr 
    lb_kind NAT
    persistence_timeout 0
    protocol TCP



    real_server 192.168.10.135 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
    real_server 192.168.10.134 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
}

2.3 configuration of keepalived backup

192.168.10.135 Modify the content of /etc/keepalived/keepalived.conf:

[root@node2 keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   router_id node1
}
vrrp_script chk_nginx {
    
    
	script "/data/sh/check_nginx.sh"
	interval 2
	weight -20
}

vrrp_instance VI_1 {
    
    
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.10.16
    }

}
virtual_server 192.168.10.16 80 {
    
    
    delay_loop 1
    lb_algo rr 
    lb_kind NAT
    persistence_timeout 0
    protocol TCP



    real_server 192.168.10.135 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
    real_server 192.168.10.134 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
}

2.4 Testing

Start the 192.168.10.134 and 192.168.10.135 keepalived services, and run the master and backup at the same time:

(1) Start three nginx services of RIP 192.168.10.134, 192.168.10.135, 192.168.10.136:

VIP is bound on MASTER 192.168.10.134:

View ipvs list:

[root@node1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.134:80            Masq    10     0          0         
  -> 192.168.10.135:80            Masq    10     0          0         
  -> 192.168.10.136:80            Masq    10     0          0     

The log shows that the ARP packet is sent:

Apr 13 16:47:24 test_kubeadm_web Keepalived_vrrp[103964]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 16:47:24 test_kubeadm_web Keepalived_vrrp[103964]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.10.16
Apr 13 16:47:24 test_kubeadm_web Keepalived_vrrp[103964]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 16:47:24 test_kubeadm_web Keepalived_vrrp[103964]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 16:47:24 test_kubeadm_web Keepalived_vrrp[103964]: Sending gratuitous ARP on ens33 for 192.168.10.16

BACKUP 192.168.10.135 The host keepalived log shows that it is in the BACKUP state:

[root@node2 keepalived]# tail /var/log/keepalived.log 
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.10.16
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Received advert with higher priority 100, ours 90
Apr 13 06:47:18 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Entering BACKUP STATE

、

(2) Stop the nginx service of a RIP 192.168.10.136

[root@node3 ~]# systemctl stop nginx
[root@node3 ~]# 

134 log:

Apr 13 16:53:51 test_kubeadm_web Keepalived_healthcheckers[103963]: TCP connection to [192.168.10.136]:80 failed.
Apr 13 16:53:54 test_kubeadm_web Keepalived_healthcheckers[103963]: TCP connection to [192.168.10.136]:80 failed.
Apr 13 16:53:54 test_kubeadm_web Keepalived_healthcheckers[103963]: Check on service [192.168.10.136]:80 failed after 1 retry.
Apr 13 16:53:54 test_kubeadm_web Keepalived_healthcheckers[103963]: Removing service [192.168.10.136]:80 from VS [192.168.10.16]:80

135 logs, consistent with MASTER:

Apr 13 06:53:52 node2 Keepalived_healthcheckers[70904]: TCP connection to [192.168.10.136]:80 failed.
Apr 13 06:53:55 node2 Keepalived_healthcheckers[70904]: TCP connection to [192.168.10.136]:80 failed.
Apr 13 06:53:55 node2 Keepalived_healthcheckers[70904]: Check on service [192.168.10.136]:80 failed after 1 retry.
Apr 13 06:53:55 node2 Keepalived_healthcheckers[70904]: Removing service [192.168.10.136]:80 from VS [192.168.10.16]:80

Check the ipvs list, 136 hosts have been automatically removed from the RIP cluster:

[root@node1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.134:80            Masq    10     0          0         
  -> 192.168.10.135:80            Masq    10     0          0        

The test request is forwarded normally:

[root@node1 ~]# curl 192.168.10.16
          this is 10.134 page
[root@node1 ~]# curl 192.168.10.16
this is 10.135 page
[root@node1 ~]# curl 192.168.10.16
          this is 10.134 page
[root@node1 ~]# curl 192.168.10.16
this is 10.135 page
[root@node1 ~]# 

(3) Stop the keepalived service of the 192.168.10.134 host

[root@node1 ~]# systemctl stop keepalived.service 
[root@node1 ~]# tail /var/log/keepalived.log 
Apr 13 16:53:54 test_kubeadm_web Keepalived_healthcheckers[103963]: Check on service [192.168.10.136]:80 failed after 1 retry.
Apr 13 16:53:54 test_kubeadm_web Keepalived_healthcheckers[103963]: Removing service [192.168.10.136]:80 from VS [192.168.10.16]:80
Apr 13 16:56:41 test_kubeadm_web Keepalived[103962]: Stopping
Apr 13 16:56:41 test_kubeadm_web Keepalived_vrrp[103964]: VRRP_Instance(VI_1) sent 0 priority
Apr 13 16:56:41 test_kubeadm_web Keepalived_vrrp[103964]: VRRP_Instance(VI_1) removing protocol VIPs.
Apr 13 16:56:41 test_kubeadm_web Keepalived_healthcheckers[103963]: Removing service [192.168.10.135]:80 from VS [192.168.10.16]:80
Apr 13 16:56:41 test_kubeadm_web Keepalived_healthcheckers[103963]: Removing service [192.168.10.134]:80 from VS [192.168.10.16]:80
Apr 13 16:56:41 test_kubeadm_web Keepalived_healthcheckers[103963]: Stopped
Apr 13 16:56:42 test_kubeadm_web Keepalived_vrrp[103964]: Stopped
Apr 13 16:56:42 test_kubeadm_web Keepalived[103962]: Stopped Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

134 There is no ipvs cluster on the host:

[root@node1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

The 135 host shows that it has switched to master:

Apr 13 06:56:42 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Transition to MASTER STATE
Apr 13 06:56:43 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Entering MASTER STATE
Apr 13 06:56:43 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) setting protocol VIPs.
Apr 13 06:56:43 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16
Apr 13 06:56:43 node2 Keepalived_vrrp[70905]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on ens33 for 192.168.10.16
Apr 13 06:56:43 node2 Keepalived_vrrp[70905]: Sending gratuitous ARP on ens33 for 192.168.10.16

View the ipvs list on 135:

[root@node2 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.134:80            Masq    10     0          0         
  -> 192.168.10.135:80            Masq    10     0          0         
[root@node2 keepalived]# 

3. Use keepalived configuration to implement DR mode

• Environment description:
  

3.1 Overview of forwarding process:

• Forwarding process:
  1. First, the user uses CIP to request VIP, and there must be a route in the middle, which is omitted here for the convenience of understanding
  .
  2. According to the above figure, we can see that VIP needs to be configured on both the scheduler and the Real Server. Then
  when a user request arrives at the front-end router of our cluster network, the source address of the request packet is CIP and the
  destination address is VIP. At this time, the switch will send a broadcast to ask who is the VIP, then all the nodes in our cluster are
  configured with VIPs. At this time, whoever responds to the router first will send the user request to whom, so that the scheduling of
  our cluster system It is meaningless, then we can make the Real Server not respond to the
  ARP address resolution request from the network, so that the user's request packet will only find the VIP of the scheduler.
  3. When the scheduler receives the user's request, it determines to load the request
  to a certain Real Server according to the result of the previously set scheduling algorithm. Assume that at this time, according to the result of the scheduling algorithm, it will load the request to
  RealServer 1. , at this time, the scheduler will modify the target MAC address in the data frame to the
  MAC address of Real Server1, and then send the data frame.
  4. When Real Server1 receives a data packet whose source address is CIP and destination address is VIP, Real
  Server1 finds that the destination address is VIP, and VIP is itself, so it accepts the data packet and processes it. After
  Real Server1 finishes processing the request, A data packet whose source address is VIP and destination address is CIP will be sent out
  .

3.2 keepalived configuration:

1) Configuration on 192.168.10.134:

[root@node1 keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   router_id node1
}
vrrp_script chk_nginx {
    
    
	script "/data/sh/check_nginx.sh"
	interval 2
	weight -20
}

vrrp_instance VI_1 {
    
    
    state MASTER
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.10.16/24
    }

}
virtual_server 192.168.10.16 80 {
    
    
    delay_loop 1
    lb_algo rr 
    lb_kind DR
    persistence_timeout 0
    protocol TCP

    real_server 192.168.10.134 80 {
    
    
        weight 10
	TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }

    real_server 192.168.10.135 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
}

2) 3.2 Configuration on 192.168.10.135:

[root@node2 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
    
    
   router_id node1
}
vrrp_script chk_nginx {
    
    
	script "/data/sh/check_nginx.sh"
	interval 2
	weight -20
}

vrrp_instance VI_1 {
    
    
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
    
    
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
    
    
        192.168.10.16
    }

}
virtual_server 192.168.10.16 80 {
    
    
    delay_loop 1
    lb_algo rr 
    lb_kind DR
    persistence_timeout 0
    protocol TCP

    real_server 192.168.10.135 80 {
    
    
        weight 10
	TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }

    real_server 192.168.10.134 80 {
    
    
        weight 10
        TCP_CHECK {
    
    
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
	    connect port 80
        }
    }
}

3.3 RIP configuration

1) Bind the VIP to the lo:1 network card, so that when rip receives the request, it finds that destIp is a VIP, and the VIP exists on its own network card, and then processes the request

2) Set arp suppression

# arp suppression:
# arp_ignore The default is 0, which means that as long as any network card device on this machine has this ip, it will respond to the arp request; set it to 1, which means that when the arp request comes, if the receiving device does not have this ip , does not respond.
# arp_announce The default is 0, which means that any address on any interface will be announced to the outside, and set to 2, which means that it will only announce to the network that matches the address on the local interface. echo 1 > /proc/sys/net/ipv4/conf
/ all/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce

The configuration of RIP can be written as the following script for one-click configuration:

[root@node2 ~]# cat /data/sh/keepalived2.sh 
#!/bin/bash
#by cuicj
#############################
VIP=192.168.10.16
RIP=192.168.10.135
INTERFACE=lo:1
GATEWAY=192.168.10.134
case "$1" in
start)
cd /etc/sysconfig/network-scripts/
cat > ifcfg-$INTERFACE <<-EOF
DEVICE=$INTERFACE
IPADDR=$VIP
NETMASK=255.255.255.255
ONBOOT=yes
NAME=loopback
EOF
ifdown $INTERFACE &>/dev/null && ifup ${INTERFACE}
echo 1 >/proc/sys/net/ipv4/conf/all/arp_ignore
echo 2 >/proc/sys/net/ipv4/conf/all/arp_announce
route add default gw $GATEWAY &>/dev/null
;;
stop)
ifdown $INTERFACE
cd /etc/sysconfig/network-scripts/
rm -rf ifcfg-$INTERFACE
route del default gw $GATEWAY &>/dev/null
;;
status)
echo "VIP: `ip a | awk '/lo:1/{print $2}'`"
echo "GATEWAY: `route -n | awk 'NR==3{print $2}'`"
;;
*)
echo "$0: Usage: $0 {start|status|stop}"
exit 1
;;
esac

Execute the scripts on 192.168.10.134 and 135 respectively. After starting keepalived, after the execution is completed, the following information will appear:
134:

135：

3.3 DR mode test:

1) Polling:
Access VIP 192.168.10.16 on client CIP 192.168.10.136

2) Automatic elimination and recovery of faulty hosts:
Stop nginx service on 134 hosts:

[root@node1 keepalived]# systemctl stop nginx
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.135:80            Route   10     0          2         
[root@node1 keepalived]# 

On 136, only the query results of 135 hosts are returned:

134 hosts restore the nginx service:

[root@node1 keepalived]# systemctl start nginx
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.134:80            Route   10     0          0         
  -> 192.168.10.135:80            Route   10     0          5         
[root@node1 keepalived]# 

The polling result is returned on 136:

3) ha high availability test:

134 The host stops the keepalived service:

[root@node1 keepalived]# systemctl stop keepalived
[root@node1 keepalived]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
[root@node1 keepalived]# 

135 takes over and becomes MASTER:

[root@node2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.10.16:80 rr
  -> 192.168.10.134:80            Route   10     0          0         
  -> 192.168.10.135:80            Route   10     0          0         
[root@node2 ~]# 

136 Test Access VIP Polling: