The deployment of load balancing LVS-DR cluster

1. LVS-DR data packet flow analysis

In order to facilitate the principle analysis, the Client and the cluster machine are placed on the same network, and the route of the data packet is 1-2-3-4.

1. The client sends a request to the target VIP, and the Director (load balancer) receives it. At this time, the source MAC address is the Client MAC address, and the destination MAC address is the MAC address of the scheduler Director.

2. The Director selects RealServer_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. At this time, the source MAC address is the MAC address of Director, and the destination MAC address is the MAC address of RealServer_1.

3. RealServer_1 receives this frame and finds that the target IP matches the local machine after decapsulation (RealServer is bound to VIP in advance), so it processes this message. Then re-encapsulate the message, and send the response message to the physical network card through the lo interface and then send it out. At this time, the source MAC address is the MAC address of RealServer_1, and the destination MAC address is the MAC address of the client.

4. The client will receive the reply message. The Client thinks that it gets the normal service, but does not know which server handles it.
Note: If it crosses the network segment, the message will be returned to the user via the Internet via the router.

Second, the ARP problem in LVS-DR

1. In the LVS-DR load balancing cluster, both the load balancing and the node server must be configured with the same VIP address.

2. Having the same IP address in the local area network will inevitably cause disorder of ARP communication between servers.

• When the ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast.
• Only the front-end load balancer responds, and other node servers should not respond to ARP broadcasts.

3. Process the node server so that it does not respond to ARP requests for VIPs.

• Use virtual interface lo:0 to carry VIP addresses
• Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP

4. RealServer returns the message (the source IP is VIP) and is forwarded by the router. When re-encapsulating the message, the MAC address of the router must be obtained first.

5. When sending an ARP request, Linux uses the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet by default, instead of using the IP address of the sending interface

• Such as: ens33

6. After the router receives the ARP request, it will update the ARP table entry

7. The MAC address of the Director corresponding to the original VIP will be updated to the MAC address of the RealServer corresponding to the VIP

8. The router forwards the new request message to RealServer according to the ARP table entry, causing the Director's VIP to fail.
Solution:

• To process the node server, set the kernel parameter arp_announce=2: the system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface.

9. Set up methods to solve the two problems of ARP
Modify the /etc/sysctl.conf file

net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Three, DR mode, LVS load balancing cluster

(1) Data packet flow analysis

(1) The client sends a request to the Director Server (load balancer), and the requested data message (source IP is CIP, destination IP is VIP) reaches the kernel space.
(2) Director Server and Real Server are in the same network, and data is transmitted through the second-layer data link layer.
(3) The kernel space judges that the target IP of the data packet is the local VIP. At this time, IPVS (IP virtual server) compares whether the service requested by the data packet is a cluster service, and repackages the data packet if it is a cluster service. Modify the source MAC address to the MAC address of the Director Server and modify the destination MAC address to the MAC address of the Real Server. The source and destination IP addresses have not changed, and then send the data packet to the Real Server.
(4) The MAC address of the request message arriving at the Real Server is its own MAC address, and the message is received. The data packet re-encapsulates the message (the source IP address is VIP and the destination IP is CIP), and the response message is transmitted to the physical network card through the lo interface and then sent out.
(5) Real Server directly transmits the response message to the client.

(2) Features of DR mode

(1) Director Server and Real Server must be in the same physical network.
(2) Real Server can use private address or public network address. If you use a public network address, you can directly access RIP through the Internet.
(3) Director Server serves as the access portal of the cluster, but not as a gateway.
(4) All request messages go through Director Server, but reply response messages cannot go through Director Server.
(5) The gateway of the Real Server is not allowed to point to the Director Server IP, that is, the data packets sent by the Real Server are not allowed to pass through the Director Server.
(6) Configure the VIP IP address on the lo interface on the Real Server.

Four, LVS-DR load balancing cluster deployment steps

Setting up environment:
DR server (load scheduler) (centos7-5): 192.168.200.50
Web server 1 (centos7-6): 192.168.200.60
Web server 2 (centos7-7): 192.168.200.70
NFS server (centos7-8) : 192.168.200.80
VIP: 192.168.200.188
Windows10 client: 192.168.200.200

1. Configure the load scheduler (192.168.200.50)

systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0
modprobe ip_vs         #加载ip_vs模块
cat /proc/net/ip_vs    #查看ip_vs版本信息
yum install -y ipvsadm

（1）、配置虚拟 IP 地址（VIP：192.168.200.188）
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens33:0

vim ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.200.188
NETMASK=255.255.255.255

ifup ens33:0
ifconfig ens33:0

（2）、调整 proc 响应参数
#由于 LVS 负载调度器和各节点需要共用 VIP 地址，应该关闭Linux 内核的重定向参数响应，不充当路由器，
vim /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

（3）、配置负载分配策略
ipvsadm-save > /etc/sysconfig/ipvsadm
或者
ipvsadm  --save > /etc/sysconfig/ipvsadm

systemctl start ipvsadm.service

ipvsadm -C     #清除原有策略
ipvsadm -A -t 192.168.200.188:80 -s rr
ipvsadm -a -t 192.168.200.188:80 -r 192.168.200.60:80 -g    #若为隧道模式，-g替换为-i
ipvsadm -a -t 192.168.200.188:80 -r 192.168.200.70:80 -g

ipvsadm -ln   #查看节点状态，Route代表 DR模式

2. Deploy shared storage (NFS server: 192.168.200.80)

systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0

yum install -y nfs-utils rpcbind

systemctl start nfs.service 
systemctl start rpcbind.service
systemctl enable nfs.service 
systemctl enable rpcbind.service

mkdir /opt/gcc /opt/benet
chmod 777 /opt/gcc/ /opt/benet/

vim /etc/exports
/usr/share *(ro,sync)
/opt/gcc 192.168.200.0/24(rw,sync)     #/24与()之间不能有空格
/opt/benet 192.168.200.0/24(rw,sync)

exportfs -rv
showmount -e

3. Configure the node server (192.168.200.60, 192.168.200.70)

systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0

#将两个节点服务器的网关和DNS注释掉后重启网卡，如果有网关服务器则指向网关服务器

（1）、配置虚拟 IP 地址（VIP：192.168.200.188）
#此地址仅用做发送 Web 响应数据包的源地址，并不需要监听客户机的访问请求（改由调度器监听并分发）。因此使用虚接口 lo:0 来承载 VIP 地址，并为本机添加一条路有记录，将访问 VIP 的数据限制在本地，以避免通信紊乱。

cd /etc/sysconfig/network-scripts/
cp ifcfg-lo ifcfg-lo:0
vim ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.200.188
NETMASK=255.255.255.255     #注意子网掩码必须全为1
#NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
#BROADCAST=127.255.255.255
ONBOOT=yes
#NAME=loopback

ifup lo:0
ifconfig lo:0

route add -host 192.168.200.188 dev lo:0    #禁锢路由
route -n    #查看路由

vim /etc/rc.local
/sbin/route add -host 192.168.200.188 dev lo:0
chmod +x /etc/rc.d/rc.local

（2）、调整内核的ARP 响应参数以阻止更新 VIP 的MAC 地址，避免发生冲突
vim /etc/sysctl.conf
......
net.ipv4.conf.lo.arp_ignore = 1   #系统只响应目的IP为本地IP的ARP请求
net.ipv4.conf.lo.arp_announce = 2  #系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

sysctl -p

或者
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

yum install -y nfs-utils rpcbind httpd
systemctl start rpcbind
systemctl start httpd


----------192.168.200.60-----------------
mount.nfs 192.168.200.80:/opt/gcc /var/www/html/
echo 'this is gcc web!' > /var/www/html/index.html

#设为自动挂载
vim /etc/fstab 
192.168.200.80:/opt/gcc /var/www/html nfs defaults,_netdev 0 0

mount -a


----------192.168.200.70-----------------
mount.nfs 192.168.200.80:/opt/benet /var/www/html/
echo 'this is benet web!' > /var/www/html/index.html

#设为自动挂载
vim /etc/fstab 
192.168.200.80:/opt/benet /var/www/html nfs defaults,_netdev 0 0

mount -a

4. Test the LVS cluster

Use a browser on the client to access: http://192.168.200.188/, the default gateway points to 192.168.200.188