1. Understand the working principle of LVS-DR
1.1 Overview of DR mode
■Load balancing cluster working mode-Direct Routing
- Referred to as DR mode, it adopts a semi-open network structure, which is similar to the structure of TUN mode, but the nodes are not scattered everywhere, but are located on the same physical network as the scheduler
- The load scheduler is connected to each node server through the local network, without the need to establish a dedicated IP tunnel
1.2. Analyze the flow of LVS-DR packets
- In order to facilitate the principle analysis, put the Client and the cluster machine on the same network, and the route of the data packet flow is 1-2-3-4
- The client sends a request to the target VIP, and the Director (load balancer) receives it. At this time, the IP header and data frame header information is
- Director selects Realserver_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. The IP header and data frame header information is as follows
- RealServer_1 receives this frame and finds that the target IP matches the machine after decapsulation (RealServer is bound to VIP in advance), so it processes the message. Then re-encapsulate the message and send it to the LAN. At this time, the IP header and data frame header information is
- Client will receive the reply message. Client thinks that it is getting normal service, but does not know which server handles it
Note: If it crosses the network segment, the message will be returned to the user via the router via terne
1.3, ARP problems in LVS-DR
- In the LVS-DR load balancing cluster, the load balancer and node server must be configured with the same VIP address
■Having the same IP address in the local area network will inevitably cause the disorder of the ARP communication of each server
- When an ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast
- At this time, only the front-end load balancer responds, and other node servers should not respond to ARP broadcasts
■Process the node server so that it does not respond to ARP requests for VIP
- Use virtual interface lo:0 to carry VIP address
- Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP
■RealServe return packets (source IP is VIP) are forwarded by the router, and the MAC address of the router must be obtained first when re-encapsulating the packet
- When sending an ARP request, Linux defaults to use the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet instead of the IP address of the sending interface (such as ens33)
- After the router receives the ARP request, it will update the ARP table entry
- The original VIP corresponding to the Director's MAC address will be updated to the ⅥP corresponding to the MAC address of the RealServer
- At this time, the new request message, the router will forward the message to RealServer according to the ARP table entry, which will cause the Director’s VIP to fail
- Solution
Process the node server and set the kernel parameter arp_announce=2: The system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface
1.4. Methods to solve the above two ARP problems
■Modify the /etc/sysctl.conf file
- Process the node server so that it does not respond to ARP requests for VIP
- net.ipv4.conf.lo.arp_ignore = 1
- net.ipv4.conf.lo.arp_announce = 2
- The system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface
- net.ipv4.conf.all.arp_ignore = 1
- net.ipv4.conf.all.arp_announce = 2
2. LVS-DR experimental deployment
2.1. Environment deployment
| scheduler | VM1： 192.168.100.21 | ens3: 0 ： 192.168.100.10 |
|---|---|---|
| WEB1 server cluster pool | VM1： 192.168.100.22 | lo: 0 ： 192.168.100.10 |
| WEB2 server cluster pool | VM1： 192.168.100.23 | lo: 0 ： 192.168.100.10 |
| NFS shared server | VM1： 192.168.100.24 |
2.2, virtual machine configuration
2.2.1, scheduler configuration
- Configure virtual IP address (VIP)
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:0
[root@localhost network-scripts]# vi ifcfg-ens33:0
NAME=ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.100.10
NETMASK=255.255.255.255
[root@localhost network-scripts]# ifup ifcfg-ens33:0
[root@localhost network-scripts]# ifconfig ## 如果-bash: ifconfig: command not found报错，安装 sudo yum install net-tools，按提示输入y，就可以了
- Adjusting the /proc response parameters For the DR cluster mode, since the LVS load scheduler and each node need to share the VIP address, the redirection parameters of the Linux kernel should be turned off. The response
server is not a router, so it will not send redirection, so it can Turn off this feature
[root@localhost network-scripts]# vi /etc/sysctl.conf
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
[root@localhost network-scripts]# sysctl -p ###生效
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
- Configure load distribution strategy
[root@localhost /]# ipvsadm -v
[root@localhost ~]# modprobe ip_vs
[root@localhost ~]# cat /proc/net/ip_vs
[root@localhost ~]#yum -y install ipvsadm
[root@localhost ~]#ipvsadm -A -t 192.168.100.10:80 -s rr
[root@localhost ~]#ipvsadm -a -t 192.168.100.10:80 -r 192.168.100.22:80 -g -w 1
[root@localhost ~]#ipvsadm -a -t 192.168.100.10:80 -r 192.168.100.23:80 -g -w 1
[root@localhost network-scripts]# ipvsadm-save
[root@localhost network-scripts]# systemctl enable ipvsadm
2.2.2, storage server
rpm -q nfs-utils ###如果没装，yum -y install nfs-utils
rpm -q rpcbind ###如果没装，yum -y install rpcbind
[root@localhost ~]# systemctl start nfs
[root@localhost ~]# systemctl start rpcbind
[root@localhost ~]# vi /etc/exports
/opt/51xit 192.168.100.0/24 (rw,sync)
/opt/52xit 192.168.100.0/24 (rw,sync)
[root@localhost ~]# systemctl restart nfs
[root@localhost ~]# systemctl restart rpcbind
[root@localhost ~]# systemctl enable nfs
[root@localhost ~]# systemctl enable rpcbind
[root@localhost ~]# mkdir /opt/51xit /opt/52xit
[root@localhost ~]# echo "this is 51xit" >/opt/51xit/index.html
[root@localhost ~]# echo "this is 52xit" >/opt/52xit/index.html
2.2.3, WEB1 server
- Configure virtual IP address
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.100.10
NETMASK=255.255.255.255
ONBOOT=yes
[root@localhost network-scripts]# ifup lo:0
[root@localhost network-scripts]# ifconfig ## 如果-bash: ifconfig: command not found报错，安装 sudo yum install net-tools，按提示输入y，就可以了
[root@localhost network-scripts]# vi /etc/rc.local
/sbin/route add -host 192.168.100.10 dev lo:0
[root@localhost network-scripts]# route add -host 192.168.100.10 dev lo:0
- Adjust /proc response parameters
[root@localhost network-scripts]# vi /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
- Install httpd mount test page
[root@localhost ~]# yum -y install nfs-utils
[root@localhost ~]# showmount -e 192.168.100.24 ####如果还没发布，请到存储服务器发布下，exportfs -rv
Export list for 192.168.100.24:
/opt/51xit (everyone)
/opt/52xit (everyone)
[root@localhost ~]# yum -y install httpd
[root@localhost ~]# mount 192.168.100.24:/opt/51xit /var/www/html/
[root@localhost ~]# vi /etc/fstab
192.168.100.44:/opt/51xit/ /var/www/html/ nfs rw,tcp,intr 0 1 ###开机自动挂载，注意格式对齐
[root@localhost ~]# systemctl start httpd
[root@localhost ~]# systemctl enable httpd
- Log in to 192.168.100.22 to test whether the website is normal
2.2.4, WEB2 server
- Configure virtual IP address
[root@localhost ~]# cd /etc/sysconfig/network-scripts/
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:0
[root@localhost network-scripts]# vi ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.100.10
NETMASK=255.255.255.255
ONBOOT=yes
[root@localhost network-scripts]# ifup lo:0
[root@localhost network-scripts]# ifconfig ## 如果-bash: ifconfig: command not found报错，安装 sudo yum install net-tools，按提示输入y，就可以了
[root@localhost network-scripts]# vi /etc/rc.local
/sbin/route add -host 192.168.100.10 dev lo:0
[root@localhost network-scripts]# route add -host 192.168.100.10 dev lo:0
- Adjust /proc response parameters
[root@localhost network-scripts]# vi /etc/sysctl.conf
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
- Install httpd mount test page
[root@localhost ~]# yum -y install nfs-utils
[root@localhost ~]# showmount -e 192.168.100.24 ####如果还没发布，请到存储服务器发布下，exportfs -rv
Export list for 192.168.100.24:
/opt/accp (everyone)
/opt/bdqn (everyone)
[root@localhost ~]# yum -y install httpd
[root@localhost ~]# mount 192.168.100.24:/opt/52xit /var/www/html/
[root@localhost ~]# vi /etc/fstab
192.168.100.24:/opt/52xit/ /var/www/html/ nfs rw,tcp,intr 0 1 ###开机自动挂载，注意格式对齐
[root@localhost ~]# systemctl start httpd
[root@localhost ~]# systemctl enable httpd
- Log in to 192.168.100.23 to test whether the website is normal
- Enter 192.168.100.10 in the browser to verify the
polling is successful
