LVS-DR cluster

LVS-DR cluster

1. Data packet flow analysis

• (1) The client sends a request to the Director Server (load balancer), and the requested data message (source IP address is CIP, destination IP address is VIP) arrives in the kernel space.

• (2) Director Server and Real Server are in the same network, and the data is transmitted through the two-layer data link layer.

• (3) The internal space judges that the target IP address of the data packet is the local VIP. At this time, IPVS (IP virtual server) compares whether the packet service is a cluster service, and repackages the data packet if it is a cluster service. Modify the source MAC address to the MAC address of Directo Sertver, modify the target MAC address to the MAC address of Real Server, the source IP address and the target IP address have not changed,

• (4) The MA address of the request message arriving at the Real Server is its own MAC address, and the message is received. The data packet re-encapsulates the message (the source IP address is the IP address, and the destination IP address is the VIP), and the response message is transmitted to the physical network card through the lo interface, and then sent out.

• (5) The Real server directly transmits the response message to the client.

2. Features of DR mode

• (1) Director server and Real server must be in the same physical network.

• (2) Real Server can use private address or public address. If the public network address is used, RIP can be directly accessed through the Internet.

• (3) Director server is used as the access entrance of the cluster, but not as a gateway.

• (4) All request messages pass through the Director server, but reply response messages cannot pass through the Director server.

• (5) The gateway of the Real server is not allowed to point to the Director server IP, that is, the data packets sent by the Real server are not allowed to pass through the Director Server.

• (6) Configure the IP address of the VIP on the lo interface on the Real server.

3. ARP parsing problem

3.1 Problem 1: IP address conflict

Problem Description

When a user on the public network sends a request to the router, it must be the load balancer server instead of the real server 1.

There will be a problem here, because when the intranet uses ARP (broadcast) to query the target MAC address when forwarding data frames, the target IP address is the VIP address (that is, in the cluster of the DR load balancing server). Both the load balancing server and the node server must be configured with the same vip address, so there will be confusion here.

If the ARP resolution at this time is the real server 1, then the service requests coming in from the external network will not pass through the load balancer but will be directly sent to the real server, so that the effect of load balancing cannot be achieved, so how do we solve it?

solution

Then we only need to let the real server 1 not respond when the ARP responds. Use the virtual interface lo:0 to carry the VIP address, set the kernel parameter arp_ipnore=1, and the system request only responds to the ARP request whose destination IP address is the local IP address.

The packet returned by RealServer (the source IP address is the VIP address) is forwarded by the router. When repackaging the packet, the MAC address of the router needs to be obtained first. When sending an ARP request, Linux uses the source IP address of the IP packet (ie VIP) as the source IP address in the ARPi request packet by default, instead of using the IP address of the sending interface (such as: ens33 interface).

After the router receives the ARP request, it will update the ARP entry, and the original VIP corresponding to the MACA address table of the Director will be updated to the MAC address of the VIP corresponding to the RealServer.

3.2 Question 2: The second ARP request of the real server

Problem Description

When the real server wants to send the data frame to the router, the source ip address becomes the virtual ip address, and the destination ip address becomes the user's ip address, which is actually just a source-to-destination conversion.

According to the three-layer forwarding principle, it can be known that the data frame is forwarded by changing the MAC address, so the real server must pass through the router if it wants to send the returned information to the user. The real server of the router's ip address must know it, because it is on the same network segment, but it does not know the router's MAC address, then it will make a second ARP request (broadcast).

At this time, the source address of the ARP broadcast is lo：0the IP address (VIP address), and the source MAC address of the real server will be brought when flooding, because we finally let the router recognize it when the first ARP request is made. The MAC address on the load balancer, when the real server makes the second ARP request, the router will find that the ip address has been recorded, and here comes another same request, but the MAC address is inconsistent, so it needs to be updated. The MAC address corresponding to the original VIP address will be updated to the MAC address of the real server, then the request sent from the client will be sent to the real server again without going through the load balancer, resulting in load balancing failure.

Solution

Process the node server, modify its kernel, so that the system does not use the source address of the ip packet to set the source address of the ARP request, but chooses the ip address of the sending interface, that is, the data frame first goes out from lo:0 to ens33 and then goes to Router, after setting this kernel information, it will no longer use the virtual ip address of lo:0, but use the real ip address of ens33 to send to the router. In this way, it will not conflict with the load-balanced VIP, and the real server will Know how to get to the router.

3.3 Setting method to solve two problems of ARP

modify /etc/systctl.conffile

net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_ignore = 1

The server only responds to the ARP request message whose destination address is the ipd address of the physical network card.

net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2

Make the server use the IP address of the network card that sends the message as the source IP address of the ARP request message.

4. DR mode LVS load balancing cluster deployment

DR 服务器：192.168.154.13
Web 服务器1：192.168.154.14
Web 服务器2：192.168.154.15
vip：192.168.154.10
客户端：192.168.154.11

1) Configure load scheduler (192.168.154.13)

systemctl stop firewalld.service
setenforce 0
modprobe ip_vs
cat /proc/net/ip_vs
yum -y install ipvsadm

（1）配置虚拟 IP 地址（VIP：192.168.154.10）
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens33:0				#若隧道模式，复制为ifcfg-tunl0
vim ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.80.188
NETMASK=255.255.255.255

ifup ens33:0
ifconfig ens33:0


（2）调整 proc 响应参数
#由于 LVS 负载调度器和各节点需要共用 VIP 地址，需要关闭 icmp 的重定向，不充当路由器。
vim /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

（3）配置负载分配策略
ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm

ipvsadm -C
ipvsadm -A -t 192.168.80.188:80 -s rr
ipvsadm -a -t 192.168.80.188:80 -r 192.168.80.12:80 -g			#若隧道模式，-g替换为-i
ipvsadm -a -t 192.168.80.188:80 -r 192.168.80.13:80 -g
ipvsadm

ipvsadm -ln					#查看节点状态，Route代表 DR模式

2) Deploy shared storage (NFS server: 192.168.80.12)

systemctl stop firewalld.service
setenforce 0

yum -y install nfs-utils rpcbind
mkdir /opt/kgc /opt/benet
chmod 777 /opt/kgc /opt/benet

vim /etc/exports
/usr/share *(ro,sync)
/opt/kgc 192.168.80.0/24(rw,sync)
/opt/benet 192.168.80.0/24(rw,sync)

systemctl start rpcbind.service
systemctl start nfs.service

3) Configure the node server (192.168.154.14, 192.168.154.15)

systemctl stop firewalld.service
setenforce 0

（1）配置虚拟 IP 地址（VIP：192.168.154.10）
#此地址仅用作发送 Web响应数据包的源地址，并不需要监听客户机的访问请求（改由调度器监听并分发）。因此使用虚接口 lo∶0 来承载 VIP 地址，并为本机添加一条路由记录，将访问 VIP 的数据限制在本地，以避免通信紊乱。
cd /etc/sysconfig/network-scripts/
cp ifcfg-lo ifcfg-lo:0		
vim ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.80.188
NETMASK=255.255.255.255						#注意：子网掩码必须全为 1

ifup lo:0
ifconfig lo:0
route add -host 192.168.80.188 dev lo:0

vim /etc/rc.local
/sbin/route add -host 192.168.80.188 dev lo:0：

chmod +x /etc/rc.d/rc.local

(2) Adjust the ARP response parameters of the kernel to prevent updating the MAC address of the VIP to avoid conflicts

vim /etc/sysctl.conf
......
net.ipv4.conf.lo.arp_ignore = 1			#系统只响应目的IP为本地IP的ARP请求
net.ipv4.conf.lo.arp_announce = 2		#系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

sysctl -p

或者
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

sysctl -p

yum -y install nfs-utils rpcbind httpd
systemctl start rpcbind
systemctl start httpd

--192.168.154.14---
mount.nfs 192.168.80.13:/opt/kgc /var/www/html
echo 'this is kgc web!' > /var/www/html/index.html

--192.168.154.15---
mount.nfs 192.168.80.13:/opt/benet /var/www/html
echo 'this is benet web!' > /var/www/html/index.html

4) Test the LVS cluster
Use a browser to access http://192.168.154.11/ on the client side


or test it in a virtual machine. Open a new virtual machine and use the following command

curl 192.168.154.10