LVS load balancing cluster (LVS-DR theory + actual deployment)

1. The working principle of LVS-DR

1.1 Overview of LVS-DR mode

Load balancing cluster working mode-Direct Routing is
referred to as DR mode. It adopts a semi-open network structure, which is similar to the structure of the TUN mode, but the nodes are not scattered everywhere, but are located on the same physical network as the scheduler
The load scheduler is connected to each node server through the local network, without the need to establish a dedicated IP tunnel

1.2. Analyze the flow of LVS-DR packets

(1) In
order to facilitate the principle analysis, put the Client and the cluster machine on the same network, the route of the data packet flow is 1-2-3-4

(2) The
client sends a request to the target VIP, Director (load balancer) receive. At this time, the IP header and data frame header information is:

(3)
Director selects Realserver_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then on the LAN send. The IP header and data frame header information is as follows:

(4)
RealServer_1 receives this frame and finds that the target IP matches this machine after decapsulation (RealServer is bound to VIP in advance), so it processes this message. Then re-encapsulate the message and send it to the LAN. At this time, the IP header and data frame header information is:

(5)
Client will receive the reply message. Client thinks that it is getting normal service, but does not know which server handles it

Note: If it crosses the network segment, the message will be returned to the user via the router via terne

1.3 ARP problems in LVS-DR

(1)
In the LVS-DR load balancing cluster, the load balancer and the node server must be configured with the same VIP address

(2)
Having the same IP address in the local area network will inevitably cause disorder in the ARP communication of each server

When an ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast.
At this time, only the front-end load balancer responds, and other node servers do not. Should respond to ARP broadcast

(3)
Process the node server so that it does not respond to ARP requests for VIP

Use the virtual interface lo:0 to carry the VIP address.
Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP

(4)
RealServe returns the packet (the source IP is VIP) and is forwarded by the router. When re-encapsulating the packet, you need to obtain the router's MAC address first. When
sending an ARP request, Linux uses the source IP address of the IP packet (namely VIP) as the ARP by default. The source IP address in the request packet instead of the IP address of the sending interface (for example, ens33)

(5) After the
router receives the ARP request, it will update the ARP table entry

The original VIP corresponding to the Director’s MAC address will be updated to the MAC address of the ⅥP corresponding to the RealServer.


(6)
At this time, the router will forward the message to RealServer according to the ARP table entry, which will cause the Director’s VIP invalidation

1.4. Solution

Modify the /etc/sysctl.conf file
to process the node server so that it does not respond to ARP requests for VIP
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2 The
system does not use IP packets To set the source address of the ARP request, select the IP address of the sending interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

2. LVS-DR actual deployment

2.1. Experimental environment

VMware software
A centos7 is used as LVS and uses a virtual interface. The ip addresses are 192.168.100.21; 192.168.100.100

Two centos7 as Apache servers, the ip addresses are 192.168.100.22; 192.168.100.23

A centos7 is used as NFS storage, the ip address is 192.168.100.24

Note: You can leave a message if there is a problem with the ip address network configuration

2.2. Experimental purpose

The real machine visits the URL of 192.168.100.100, uses the DR mode to drift the address, and polls access to the Apache1 and Apache2 hosts.
Build an nfs network file storage service and experiment with load balancing

2.3. Experimental process

2.3.1, configure load scheduler

2.3.1.1. Configure virtual IP address (VIP)

[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[root@localhost network-scripts]# vi ifcfg-ens33:0
NAME=ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.100.100
NETMASK=255.255.255.255
[root@localhost network-scripts]# ifup ifcfg-ens33:0
[root@localhost network-scripts]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.21  netmask 255.255.255.0  broadcast 192.168.100.255
        inet6 fe80::3069:1a3d:774b:18f  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:11:0d:16  txqueuelen 1000  (Ethernet)
        RX packets 1170  bytes 110973 (108.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 683  bytes 81523 (79.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.100  netmask 255.255.255.255  broadcast 192.168.100.100
        ether 00:0c:29:11:0d:16  txqueuelen 1000  (Ethernet)

        省略部分内容

2.3.1.2, adjust /proc response parameters

For the DR cluster mode, since the LVS load scheduler and each node need to share the VIP address, the redirection parameter response of the Linux kernel should be turned off. The
server is not a router, so it will not send redirection, so this function can be turned off

[root@localhost network-scripts]# vi /etc/sysctl.conf 
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

2.3.1.3, configure load distribution strategy

[root@localhost ~]#yum -y install ipvsadm
[root@localhost /]# ipvsadm -v
[root@localhost ~]# modprobe ip_vs
[root@localhost ~]# cat /proc/net/ip_vs
[root@localhost ~]# ipvsadm -A -t 192.168.100.100:80 -s rr
[root@localhost ~]# ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.22:80 -g -w 1
[root@localhost ~]# ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.23:80 -g -w 1
[root@localhost ~]# ipvsadm-save > /opt/ipvsadm
[root@localhost ~]# cat /opt/ipvsadm
-A -t localhost.localdomain:http -s rr
-a -t localhost.localdomain:http -r 192.168.100.22:http -g -w 1
-a -t localhost.localdomain:http -r 192.168.100.23:http -g -w 1

2.3.2, configure node server

2.3.2.1. Storage server configuration (192.168.100.24)

First, check whether nfs-utils and rpcbind are installed, if they are not installed with yum
, start the two services after installation

[root@localhost ~]# systemctl start nfs
[root@localhost ~]# systemctl start rpcbind
[root@localhost ~]# mkdir /opt/51xit /opt/52xit
[root@localhost ~]# vi /etc/exports
/opt/51xit 192.168.100.0/24(rw,sync)
/opt/52xit 192.168.100.0/24(rw,sync)
[root@localhost ~]# systemctl restart rpcbind
[root@localhost ~]# systemctl restart nfs
[root@localhost ~]# systemctl enable nfs
[root@localhost ~]# systemctl enable rpcbind
[root@localhost ~]# echo "this is www.51xit.top" > /opt/51xit/index.html
[root@localhost ~]# echo "this is www.52xit.top" > /opt/52xit/index.html

2.3.2.2. Configure virtual IP address (VIP) (192.168.100.22 and 192.168.100.23)

Both the firewall and core protection are turned off, check whether nfs-utils is installed

[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.100.100
NETMASK=255.255.255.255
ONBOOT=yes

[root@localhost network-scripts]# ifup lo:0
[root@localhost network-scripts]# ifconfig
        省略部分内容
lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 192.168.100.100  netmask 255.255.255.255
        loop  txqueuelen 1000  (Local Loopback)
        省略部分内容
[root@localhost network-scripts]# vi /etc/rc.local 
/sbin/route add -host 192.168.100.100 dev lo:0

[root@localhost network-scripts]# route add -host 192.168.100.100 dev lo:0     

2.3.2.3, adjust /proc response parameters

[root@localhost network-scripts]# vi /etc/sysctl.conf 
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2

[root@localhost network-scripts]# sysctl -p

The two node servers previously configured are the same

2.3.2.4, install httpd mount test page

Mount two node servers separately below

[root@localhost ~]# showmount -e 192.168.100.24
Export list for 192.168.100.24:
/opt/52xit 192.168.100.0/24
/opt/51xit 192.168.100.0/24
[root@localhost ~]# yum -y install httpd
[root@localhost ~]# mount 192.168.100.24:/opt/51xit /var/www/html/
[root@localhost ~]# vi /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Thu Aug  6 12:23:03 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=a1c935eb-f211-43a5-be35-2a9fef1f6a89 /boot                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0
/dev/cdrom /mnt iso9660 defaults 0 0
192.168.100.24:/opt/51xit/ /var/www/html/ nfs defaults,_netdev 0 0
[root@localhost ~]# systemctl start httpd

Test whether the login is normal

[root@localhost ~]# showmount -e 192.168.100.24
Export list for 192.168.100.24:
/opt/52xit 192.168.100.0/24
/opt/51xit 192.168.100.0/24
[root@localhost ~]# yum -y install httpd
[root@localhost ~]# mount 192.168.100.24:/opt/52xit /var/www/html/
[root@localhost ~]# vi /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Thu Aug  6 12:23:03 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=a1c935eb-f211-43a5-be35-2a9fef1f6a89 /boot                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0
/dev/cdrom /mnt iso9660 defaults 0 0
192.168.100.24:/opt/52xit/ /var/www/html/ nfs defaults,_netdev 0 0
[root@localhost ~]# systemctl start httpd

Test whether the login is normal

2.4, verification

Enter 192.168.100.100 in the browser of the real machine and

re-enter it. The

verification is successful.