LVS load balancing cluster---Detailed explanation of actual deployment in DR mode

LVS load balancing cluster-DR mode actual deployment detailed explanation

1. LVS-DR data packet flow analysis

In order to facilitate the principle analysis, put the Client and the cluster machine on the same network, and the route of the data packet flow is 1-2-3-4

1. The Client sends a request to the target VIP, and the Director (load balancer) receives it. At this time, the source MAC address is the Client MAC address, and the destination MAC address is the MAC address of the scheduler Director.

2. The Director selects RealServer_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. At this time, the source MAC address is the MAC address of Director, and the destination MAC address is the MAC address of RealServer_1.

3. RealServer_1 receives this frame and finds that the target IP matches the local machine after decapsulation (RealServer is bound to VIP in advance), so it processes this message. Then re-encapsulate the message, and send the response message to the physical network card through the lo interface and then send it out. At this time, the source MAC address is the MAC address of RealServer_1, and the destination MAC address is the MAC address of the client.

4. The client will receive the reply message. The Client thinks that it gets the normal service, but does not know which server handles it.
Note: If it crosses the network segment, the message will be returned to the user via the Internet via the router.

Second, the ARP problem in LVS-DR

1. In the LVS-DR load balancing cluster, both the load balancing and the node server must be configured with the same VIP address.

2. Having the same IP address in the local area network will inevitably cause disorder of ARP communication between servers.

• When the ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast.
• Only the front-end load balancer responds, and other node servers should not respond to ARP broadcasts.

3. Process the node server so that it does not respond to ARP requests for VIPs.

• Use virtual interface lo:0 to carry VIP addresses
• Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP

4. RealServer returns the message (the source IP is VIP) and is forwarded by the router. When re-encapsulating the message, the MAC address of the router must be obtained first.

5. When sending an ARP request, Linux uses the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet by default, instead of using the IP address of the sending interface

• Such as: ens33

6. After the router receives the ARP request, it will update the ARP table entry

7. The MAC address of the Director corresponding to the original VIP will be updated to the MAC address of the RealServer corresponding to the VIP

8. The router forwards the new request message to RealServer according to the ARP table entry, causing the Director's VIP to fail.
Solution:

• To process the node server, set the kernel parameter arp_announce=2: the system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface.

9. Set up methods to solve the two problems of ARP
Modify the /etc/sysctl.conf file

net.ipv4.conf.lo.arp_ignore=1 
net.ipv4.conf.lo.arp_announce=2 
net.ipv4.conf.all.arp_ignore=1 
net.ipv4.conf.all.arp_announce=

2

Three, DR mode, LVS load balancing cluster

(1) Data packet flow analysis

(1) The client sends a request to the Director Server (load balancer), and the requested data message (source IP is CIP, destination IP is VIP) reaches the kernel space.
(2) Director Server and Real Server are in the same network, and data is transmitted through the second-layer data link layer.
(3) The kernel space judges that the target IP of the data packet is the local VIP. At this time, IPVS (IP virtual server) compares whether the service requested by the data packet is a cluster service, and repackages the data packet if it is a cluster service. Modify the source MAC address to the MAC address of the Director Server and modify the destination MAC address to the MAC address of the Real Server. The source and destination IP addresses have not changed, and then send the data packet to the Real Server.
(4) The MAC address of the request message arriving at the Real Server is its own MAC address, and the message is received. The data packet re-encapsulates the message (the source IP address is VIP and the destination IP is CIP), and the response message is transmitted to the physical network card through the lo interface and then sent out.
(5) Real Server directly transmits the response message to the client.

(2) Features of DR mode

(1) Director Server and Real Server must be in the same physical network.
(2) Real Server can use private address or public network address. If you use a public network address, you can directly access RIP through the Internet.
(3) Director Server serves as the access portal of the cluster, but not as a gateway.
(4) All request messages go through Director Server, but reply response messages cannot go through Director Server.
(5) The gateway of the Real Server is not allowed to point to the Director Server IP, that is, the data packets sent by the Real Server are not allowed to pass through the Director Server.
(6). Configure the VIP IP address on the lo interface on the Real Server.

4. Project actual operation

(1) Experimental environment requirements

Five linux virtual machines

DR 服务器：192.168.126.10
Web 服务器1：192.168.126.20
Web 服务器2：192.168.126.30
共享服务器：192.168.126.40
客户端：192.168.126.50
vip（虚拟IP）:192.168.126.88

(Two), experimental steps

1. Configure the load scheduler (192.168.126.10)

systemctl stop firewalld.service
setenforce 0
modprobe ip_vs
cat /proc/net/ip_vs
yum -y install ipvsadm

(1), configure virtual IP address (VIP: 192.168.126.88)

cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens33:0				#若隧道模式，复制为ifcfg-tunl0
vim ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.126.88
NETMASK=255.255.255.255

ifup ens33:0        #开启

ifconfig ens33:0   # 查看虚拟ip

(2) Adjust proc response parameters

由于 LVS 负载调度器和各节点需要共用 VIP 地址，应该关闭Linux 内核的重定向参数响应。
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

(3), configure the load distribution strategy

ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm

ipvsadm -C
ipvsadm -A -t 192.168.126.88:80 -s rr
ipvsadm -a -t 192.168.126.88:80 -r 192.168.126.10:80 -g			#若隧道模式，-g替换为-i
ipvsadm -a -t 192.168.80.188:80 -r 192.168.126.20:80 -g
ipvsadm 

ipvsadm -ln					#查看节点状态，Route代表 DR模式

2. Deploy shared storage (NFS server: 192.168.126.40)

systemctl stop firewalld.service
setenforce 0

yum -y install nfs-utils rpcbind
mkdir /opt/chenwei /opt/mm
chmod 777 /opt/mm/opt/chenwei

vim /opt/chenwei/index.html   #写内容在文档里，挂载给网站首页 /var/www/html

vim /opt/mm/index.html 

vim /etc/exports
/usr/share *(ro,sync)
/opt/kgc 192.168.80.0/24(rw,sync)
/opt/benet 192.168.80.0/24(rw,sync)

systemctl start rpcbind.service
systemctl start nfs.service

3. Configure the node server (192.168.126.10, 192.168.126.20)

systemctl stop firewalld.service
setenforce 0

(1), configure virtual IP address (VIP: 192.168.126.88)

cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-lo:0		
vim ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.80.188
NETMASK=255.255.255.255						#注意：子网掩码必须全为 1

ifup lo:0
ifconfig lo:0
route add -host 192.168.80.188 dev lo:0		#添加VIP本地访问路由，将访问VIP的数据限制在本地，以避免通信紊乱

(2) Adjust proc response parameters

vim /etc/sysctl.conf
......
net.ipv4.conf.lo.arp_ignore = 1			#系统只响应目的IP为本地IP的ARP请求
net.ipv4.conf.lo.arp_announce = 2		#系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

sysctl -p

或者
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p

yum -y install nfs-utils rpcbind httpd
systemctl start rpcbind
systemctl start nfs
systemctl start httpd

--192.168.126.10---
mount.nfs 192.168.126.40:/opt/chenwei /var/www/html

--192.168.126.20---
mount.nfs 192.168.126.40:/opt/mm /var/www/html

4. Test the LVS cluster

Use a browser on the client to visit http://192.168.126.88/, the default gateway points to 192.168.126.88