keepalived role
In a large construction site, we usually use some load balancing (LVS, Nginx, Haproxy), distribute requests to the back-end services cluster.
At this time, load balancing balancing node node becomes the single point of failure, in order to ensure high availability systems can be introduced keepalived, multiple load-balancing nodes together as a whole Foreign Service, in order to prevent a single point of failure.
keepalived works
keepalived VRRP protocol is implemented on the basis of, the full name of VRRP Virtual Router Redundancy Protocol, is a fault-tolerant routing protocol, also called Standby Routing Protocol. A group of VRRP router to work together to form a virtual router. The external performance virtual router is a logical router MAC address and IP address with a unique fixed. When the actual work, a router in this group as a master node, the other as a backup node. During operation, master sends VRRP protocol packet when the master node fails, VRRP backup node can not receive the packet, wherein a backup node (priority large) will automatically become the master node, to continue to provide services, this process is completely automatic, not the need for manual intervention.
keepalived Using this principle, the different server nodes (node load balancing, web server) components into a server group, provide external uniform (VIP) address, thus ensuring high availability.
keepalived installation
$ mkdir -p /opt/k8s/keepalive-haproxy
$ cd /opt/k8s/keepalive-haproxy
$ wget https://www.keepalived.org/software/keepalived-2.0.20.tar.gz
$ tar -xzvf keepalived-2.0.20.tar.gz
$ cd /opt/k8s/keepalive-haproxy/keepalived-2.0.20
# 执行configure，prefix指定安装目录
$ mkdir -p /opt/k8s/keepalived/
$ ./configure --prefix=/opt/k8s/keepalived/
$ make & make install
# 查看安装目录
$ ls /opt/k8s/keepalived/
bin etc sbin share
A simple example
Let's binding to keepalived tomcat hot standby to achieve a web server. Network topology is as follows:
- The entire system of two nodes, nodes web1 192.168.0.114 (the primary node), node web2 192.168.0.107 (the standby node), Virtual IP (external service provider IP 192.168.0.100)
- The above two nodes are running a similar web services exposed to the 8086 port, on both nodes are installed keepalived, external access to the system through a virtual IP
- In the virtual IP actual work process at a time can belong to only one node, the other nodes as a backup node exists
- When the master node when the normal: keepalived information broadcasted on the node web1, 192.168.0.100 corresponds to the IP address of the node for MAC card web1
- Machine the same segment will update its ARP table, corresponding to the MAC address = 192.168.0.100 node's MAC address web1
- When a node fails web1, web2 keepalived on the node will be detected, and the following information is broadcasted:
192.168.8.100 the IP address corresponding to a MAC node's MAC address web2 - Other computers of the same network segment as the client updates its ARP table, corresponding to the MAC address = 192.168.8.100 node's MAC address web2
Configuration master keepalived
$ mkdir /opt/k8s/keepalived/conf $ cd /opt/k8s/keepalived/conf $ cat > keepalived.conf<<EOF global_defs { router_id web1 } vrrp_instance VI_1 { state MASTER #设置为主服务器 interface wlo1 #监测网络接口 virtual_router_id 55 #主、备必须一样 priority 200 #(主、备机取不同的优先级，主机值较大，备份机值较小,值越大优先级越高) advert_int 1 #VRRP Multicast广播周期秒数 authentication { auth_type PASS #VRRP认证方式，主备必须一致 auth_pass 1111 #(密码) } virtual_ipaddress { 192.168.0.100/24 #VRRP HA虚拟地址 } } EOFConfiguring the spare node keepalived
$ mkdir /opt/k8s/keepalived/conf $ cd /opt/k8s/keepalived/conf $ cat > keepalived.conf<<EOF global_defs { router_id web2 } vrrp_instance VI_1 { state BACKUP #设置为主服务器 interface wlp3s0 #监测网络接口 virtual_router_id 55 #主、备必须一样 priority 100 #(主、备机取不同的优先级，主机值较大，备份机值较小,值越大优先级越高) advert_int 1 #VRRP Multicast广播周期秒数 authentication { auth_type PASS #VRRP认证方式，主备必须一致 auth_pass 1111 #(密码) } virtual_ipaddress { 192.168.0.100/24 #VRRP HA虚拟地址 } } EOFKeepalived start service on the primary node
$ /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.confView Log
The master node (Ubuntu system in syslog)
$ tail -f /var/log/syslog | grep Keepalived Mar 13 17:21:42 slave Keepalived[11843]: Starting Keepalived v2.0.20 (01/22,2020) Mar 13 17:21:42 slave Keepalived[11843]: Running on Linux 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 (built for Linux 4.15.18) Mar 13 17:21:42 slave Keepalived[11843]: Command line: '/opt/k8s/keepalived/sbin/keepalived' '-f' '/opt/k8s/keepalived/conf/keepalived.conf' Mar 13 17:21:42 slave Keepalived[11843]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:21:42 slave Keepalived[11843]: Remove a zombie pid file /run/keepalived.pid Mar 13 17:21:42 slave Keepalived[11844]: Starting VRRP child process, pid=11845 Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering Kernel netlink reflector Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering Kernel netlink command channel Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:21:42 slave Keepalived_vrrp[11845]: Registering gratuitous ARP shared channel Mar 13 17:21:42 slave Keepalived_vrrp[11845]: (VI_1) Entering BACKUP STATE (init) Mar 13 17:21:45 slave Keepalived_vrrp[11845]: (VI_1) Entering MASTER STATEFrom node
$ tail -f /var/log/syslog | grep Keepalived Mar 13 17:24:30 master Keepalived[3118]: Starting Keepalived v2.0.20 (01/22,2020) Mar 13 17:24:30 master Keepalived[3118]: Running on Linux 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 (built for Linux 4.15.18) Mar 13 17:24:30 master Keepalived[3118]: Command line: '/opt/k8s/keepalived/sbin/keepalived' '-f' '/opt/k8s/keepalived/conf/keepalived.conf' Mar 13 17:24:30 master Keepalived[3118]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:24:30 master Keepalived[3118]: Remove a zombie pid file /run/keepalived.pid Mar 13 17:24:30 master Keepalived[3120]: Starting VRRP child process, pid=3122 Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering Kernel netlink reflector Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering Kernel netlink command channel Mar 13 17:24:30 master Keepalived_vrrp[3122]: Opening file '/opt/k8s/keepalived/conf/keepalived.conf'. Mar 13 17:24:30 master Keepalived_vrrp[3122]: Registering gratuitous ARP shared channel Mar 13 17:24:30 master Keepalived_vrrp[3122]: (VI_1) Entering BACKUP STATE (init)Check the master node network equipment
$ ip addr show wlo1 3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether c4:8e:8f:d0:33:89 brd ff:ff:ff:ff:ff:ff inet 192.168.0.114/24 brd 192.168.0.255 scope global dynamic noprefixroute wlo1 valid_lft 4948sec preferred_lft 4948sec inet 192.168.0.100/24 scope global secondary wlo1 valid_lft forever preferred_lft forever inet6 fe80::db1e:449b:5431:7392/64 scope link noprefixroute valid_lft forever preferred_lft forever- We can see more than one network card wlo1 the following address 192.168.0.100/24
Deploy web services
Spring boot constructed based on an application by embeded-tomcat, which provides a restful interface ports are arranged corresponding to the exposed 8086
@RequestMapping("/health") public String health(HttpServletRequest request) { return "OK"; }On the primary node startup spring boot application
Virtual IP access through
$ curl http://192.168.0.100:8086/health OK
Analog failure and recovery
Keepalived stopped the process of the master node
$ ps -elf | grep -e keepa -e PID F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 1 S root 11844 1 0 80 0 - 4188 ep_pol 17:21 ? 00:00:00 /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.conf 5 S root 11845 11844 0 80 0 - 4188 ep_pol 17:21 ? 00:00:00 /opt/k8s/keepalived/sbin/keepalived -f /opt/k8s/keepalived/conf/keepalived.conf 0 S root 19389 16663 0 80 0 - 5383 pipe_w 18:06 pts/0 00:00:00 grep --color=auto -e keepa -e PID $ kill -9 11844View from the master node network equipment
The master node
$ ip addr show wlo1 3: wlo1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether c4:8e:8f:d0:33:89 brd ff:ff:ff:ff:ff:ff inet 192.168.0.114/24 brd 192.168.0.255 scope global dynamic noprefixroute wlo1 valid_lft 5707sec preferred_lft 5707sec inet6 fe80::db1e:449b:5431:7392/64 scope link noprefixroute valid_lft forever preferred_lft foreverFrom node
$ ip addr show wlp3s0 3: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d0:c5:d3:57:73:01 brd ff:ff:ff:ff:ff:ff inet 192.168.0.107/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp3s0 valid_lft 4597sec preferred_lft 4597sec inet 192.168.0.100/24 scope global secondary wlp3s0 valid_lft forever preferred_lft forever inet6 fe80::1fda:e90a:207a:67e4/64 scope link noprefixroute valid_lft forever preferred_lft forever- You can see the virtual IP to drift from a node on the network card
Virtual IP access through
$ curl http://192.168.0.100:8086/health OK- The normal response service
Keepalived restart process on web1, see wlo1 card on the primary node again, you will find a virtual IP and automatic drift back
Script detection
The previous section we fall down through simulation keepalived process to verify the VIP automatically migrated. But when keepalived functions on the primary node is normal, but just a web service failure occurs, follow the simple configuration before, VIP is not automatically migrated, then the service becomes inaccessible up. The solution is added vrrp_script in keepalived profile
Examples are as follows:
Write detection script
$ mkdir /opt/k8s/keepalived/script $ cd /opt/k8s/keepalived/script $ cat > checkproxy.sh<<EOF #!/bin/bash count = `ps aux | grep -v grep | grep haproxy | wc -l` if [ $count > 0 ]; then exit 0 else exit 1 fiAdditionally we detected in the configuration file
... vrrp_script checkhaproxy { script "/home/checkproxy.sh" interval 3 weight -150 } vrrp_instance test { ... track_script { checkhaproxy } ... }- About check of weight
- In that the detection is successful vrrp_script script returns 0, other values are detected as a failure;
- When the weight is positive, the script detects that the success of this weight will be added to the priority, without failure is detected;
- Failure Main: main priority <priority + weight will be switched from.
- Main success: the main priority + weight> when priority + weight from the master is still the main
- Failure Main: main priority <priority + weight will be switched from.
- When the weight is negative, the script detects that the success of this weight does not affect the priority, upon detection of failure priority - abs (weight)
- Failure Main: main priority - abs (weight) <priority will be switched from the main
- Main success: the main priority> From the main priority is still the main