Theory + experiment-how to build LVS-DR load balancing cluster

theory:

1. The working principle of LVS-DR

1.1 Overview of LVS-DR mode

Load balancing cluster working mode-Direct Routing is
referred to as DR mode. It adopts a semi-open network structure, which is similar to the structure of the TUN mode, but the nodes are not scattered everywhere, but are located on the same physical network as the scheduler
The load scheduler is connected to each node server through the local network, without the need to establish a dedicated IP tunnel

1.2 Analyze the flow of LVS-DR packets

(1) In order to facilitate the principle analysis, put the Client and the cluster machine on the same network, and the route of the data packet flow is 1-2-3-4

(2) The client sends a request to the target VIP, Director (load balancer) receive. The P header and data frame header information are:

(3) Director selects Realserver_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. The IP header and data frame header information is as follows:

(4) RealServer_1 receives this frame and finds that the target IP matches the machine after decapsulation ( RealServer is bound to VIP in advance ), so it processes this message. Then re-encapsulate the message and send it to the LAN. At this time, the IP header and data frame header information is:

(5) Client will receive the reply message. Client thinks that it has received normal service, but will not know which server is processing it.
Note: If it crosses the network segment, then the message will be returned to the user via the router via terne

1.3 ARP problems in LVS-DR

(1) In the LVS-DR load balancing cluster, both the load balancer and the node server must be configured with the same VIP address

(2) The node server is processed so that it does not respond to ARP requests for VIP

• Use virtual interface lo:0 to carry VIP address
• Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP

（3）

• RealServe return packets (the source IP is VIP) are forwarded by the router, and the MAC address of the router must be obtained first when re-encapsulating the packet
• When sending an ARP request, Linux uses the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet by default, instead of using the IP address of the sending interface,
  such as: ens33

（4）

• After the router receives the ARP request, it will update the ARP table entry
• The original VIP corresponding to the Director's MAC address will be updated to the ⅥP corresponding to the MAC address of the RealServer
  

(5) Problem

• According to the ARP table entry, the router forwards the new request message to RealServer, causing the VIP of Director to become invalid
  
• To process the contact server, set the kernel parameter arp_announce=2: the system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface

1.4 How to solve the two problems of ARP

Modify the /etc/sysctl.conf file
to process the node server so that it does not respond to ARP requests for VIP
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2 The
system does not use IP packets To set the source address of the ARP request, select the IP address of the sending interface
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

experiment:

1. Experiment preparation

One scheduler: VM1: 192.168.100:21
Two WEB server cluster pools: 192.168.100.22, 192.168.100.23
One NFS shared server: 192.168.100.24

Second, the steps

2.1 Scheduler settings (192.168.100.21)

2.1.1 Configure virtual IP address (VIP)

[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[root@localhost network-scripts]# vi ifcfg-ens33:0
NAME=ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.100.100
NETMASK=255.255.255.255
[root@localhost network-scripts]# ifup ifcfg-ens33:0
[root@localhost network-scripts]# ifconfig

2.1.2 Adjust /proc response parameters

For the DR cluster mode, since the LVS load scheduler and each node need to share the VIP address, the redirection parameter response of the Linux kernel should be turned off. The
server is not a router, so it will not send redirection, so this function can be turned off

[root@localhost network-scripts]# vi /etc/sysctl.conf 
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@localhost network-scripts]# sysctl -p           ###生效
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

2.1.3 Configure load distribution strategy

[root@localhost ~]#yum -y install ipvsadm
[root@localhost /]# ipvsadm -v
[root@localhost ~]# modprobe ip_vs
[root@localhost ~]# cat /proc/net/ip_vs
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@localhost ~]#ipvsadm -A -t 192.168.100.100:80 -s rr
[root@localhost ~]#ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.42:80 -g -w 1
[root@localhost ~]#ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.43:80 -g -w 1
[root@localhost network-scripts]# ipvsadm-save > /opt/ipvsadm
[root@localhost network-scripts]# cat /opt/ipvsadm 
-A -t localhost.localdomain:http -s rr
-a -t localhost.localdomain:http -r 192.168.100.22:http -g -w 1
-a -t localhost.localdomain:http -r 192.168.100.23:http -g -w 1

2.2 Configure the storage server (192.168.100.24)

rpm -q nfs-utils    ###如果没装，yum -y install nfs-utils
rpm -q rpcbind      ###如果没装，yum -y install rpcbind
[root@localhost ~]# systemctl start nfs
[root@localhost ~]# systemctl start rpcbind
[root@localhost ~]# vi /etc/exports
/opt/51xit 192.168.100.0/24 (rw,sync)
/opt/52xit 192.168.100.0/24 (rw,sync)
'###重启需要想开启rpcbind###'
[root@localhost ~]# systemctl restart rpcbind            
[root@localhost ~]# systemctl restart nfs
[root@localhost ~]# systemctl enable nfs
[root@localhost ~]# systemctl enable rpcbind
[root@localhost ~]# mkdir /opt/51xit /opt/52xit
[root@localhost ~]# echo "this is 51xit" >/opt/51xit/index.html
[root@localhost ~]# echo "this is 52xit" >/opt/52xit/index.html

2.3 Configure node 1 server (192.168.100.22)

2.3.1 Configure virtual IP address

[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.100.10
NETMASK=255.255.255.255
ONBOOT=yes
[root@localhost network-scripts]# ifup lo:0
[root@localhost network-scripts]# ifconfig

'###添加VIP本地访问路由###'
[root@localhost network-scripts]# vi /etc/rc.local
/sbin/route add -host 192.168.100.10 dev lo:0
[root@localhost network-scripts]# route add -host 192.168.100.10 dev lo:0

2.3.2 Adjust /proc response parameters

[root@localhost network-scripts]# vi /etc/sysctl.conf 
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@localhost network-scripts]# sysctl -p

2.3.3 Install httpd mount test page

[root@localhost ~]# showmount -e 192.168.100.44     ####如果还没发布，请到存储服务器发布下，exportfs -rv
Export list for 192.168.100.24:
/opt/52xit 192.168.100.0/24
/opt/51xit 192.168.100.0/24
[root@localhost ~]# yum -y install httpd
[root@localhost ~]# mount 192.168.100.44:/opt/51xit /var/www/html/
[root@localhost ~]# vi /etc/fstab 
192.168.100.44:/opt/51xit/ /var/www/html/ nfs defaults,_netdev 0 0       ###开机自动挂载，注意格式对齐
[root@localhost ~]# systemctl start httpd
[root@localhost ~]# systemctl enable httpd

2.4 Configure node 2 server (192.168.100.23)

For the configuration of 192.168.100.22, please refer to 2.3

2.5 Testing

Enter the following page:
192.168.100.22

page, enter the following:
192.168.100.23

page enter the following:
192.168.100.100