DR mode LVS load balancing cluster (detailed pictures and texts!)

DR mode LVS load balancing cluster (detailed pictures and texts!)

1. The working principle of LVS-DR

1. According to packet flow analysis

In order to facilitate the principle analysis, put the Client and the cluster machine on the same network, and the route of the data packet flow is 1-2-3-4

1) The Client sends a request to the target VIP, and the Director (load balancer) receives it. At this time, the source MAC address is the Client MAC address, the destination MAC address is the MAC address of the scheduler Director, the source IP is the client's IP, and the destination IP is the VIP.

2) The Director selects RealServer_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. At this time, the source MAC address is the MAC address of the Director, the destination MAC address is the MAC address of RealServer_1, the source IP is the client's IP, and the destination IP is the VIP

3) RealServer_1 receives this frame and finds that the target IP matches the machine after decapsulation (RealServer is bound to VIP in advance), so it processes this message. Then re-encapsulate the message, and send the response message to the physical network card through the lo interface and then send it out. At this time, the source MAC address is the MAC address of RealServer_1, and the destination MAC address is the MAC address of the client. The source IP is VIP, and the destination IP is the client's IP

4) Transmit the response message to the client through the switch and router. The client receives the reply message and gets the desired service, but it does not know which server processed it.

2. Features of DR mode

• Director Server and Real Server must be in the same physical network.
• Real Server can use a private address or a public network address. If you use a public network address, you can directly access RIP via the Internet.
• Director Server serves as the access portal for the cluster, but not as a gateway.
• All request messages go through Director Server, but reply response messages cannot go through Director Server.
• The gateway of the Real Server is not allowed to point to the Director Server IP, that is, the data packets sent by the Real Server are not allowed to pass through the Director Server.
• The lo interface on the Real Server is configured with the IP address of the VIP.

Second, the ARP problem in LVS-DR

1. In the LVS-DR load balancing cluster, both the load balancing and the node server must be configured with the same VIP address.

2. Having the same IP address in the local area network will inevitably cause disorder in the ARP communication of each server.

When the ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast.
Only the front-end load balancer responds, and other node servers should not respond to ARP broadcasts.
3. Process the node server so that it does not respond to ARP requests for VIPs.

Use the virtual interface lo:0 to carry the VIP address.
Set the kernel parameter arp_ignore=1: the system only responds to the ARP request
whose destination IP is the local IP. 4. RealServer returns the message (the source IP is VIP) and is forwarded by the router. When the message is re-encapsulated, it is required Get the MAC address of the router first.

5. When sending an ARP request, Linux defaults to use the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet instead of the IP address of the sending interface

Such as: ens33
6. After the router receives the ARP request, it will update the ARP table entry

7. The original VIP corresponding to the Director's MAC address will be updated to the VIP corresponding to the MAC address of the RealServer

8. The router forwards the new request message to RealServer according to the ARP table entry, causing the Director's VIP to become invalid.
Solution:

To process the node server, set the kernel parameter arp_announce=2: the system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface.
9. The setting method to solve the two problems of ARP
Modify the /etc/sysctl.conf file

net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

Three, LVS load balancing DR mode cluster deployment steps

Environment setup:

Host	operating system	IP address	Required tools/services
DR server (load scheduler)	CentOS7 7-2	ens33:192.168.2.4 ens33:0(VIP)：192.168.2.100	ipvsadm
NFS server	CentOS7 7-3	ens33 ： 192.168.2.5	rpcbind、nfs-utils
Web node server 1	CentOS7 7-4	192.168.2.6 lo: 0 (VIP) ： 192.168.2.100	rpcbind、nfs-utils、httpd
Web node server 2	CentOS7 7-5	192.168.2.7 lo: 0 (VIP) ： 192.168.2.100	rpcbind、nfs-utils、httpd
Client	Windows10	192.168.2.10

Note: In the same LAN, no gateway and DNS are required.

1. Deploy shared storage

NFS server: ens33: 192.168.2.5

systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0

yum -y install nfs-utils rpcbind

systemctl start rpcbind.service
systemctl start nfs.service
systemctl enable nfs.service
systemctl enable rpcbind.service

mkdir /opt/test1
mkdir /opt/test2

chmod 777 /opt/test1
chmod 777 /opt/test2

vim /etc/exports
/opt/test1 192.168.2.0/24(rw,sync)
/opt/test2 192.168.2.0/24(rw,sync)

exportfs -rv

2. Configure the node server

Web node server 1: ens33: 192.168.2.6 lo:0 (VIP): 192.168.2.100
Web node server 2: ens33: 192.168.2.7 lo:0 (VIP): 192.168.2.100
Next is the same configuration of the two servers

systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0

(1) Configure virtual IP address (VIP: 192.168.2.100)

#此地址仅用做发送 Web 响应数据包的源地址，并不需要监听客户机的访问请求（改由调度器监听并分发）。
#因此使用虚接口 lo:0 来承载 VIP 地址，并为本机添加一条路有记录，将访问 VIP 的数据限制在本地，以避免通信紊乱。

vim /etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.2.100
NETMASK=255.255.255.255
ONBOOT=yes

ifup lo:0
ifconfig lo:0

#设置临时的路由，重启失效；禁锢路由
route add -host 192.168.2.100 dev lo:0
#查看路由
route -n

#开机自动添加路由，生产环境应该用这个
vim /etc/rc.local
/sbin/route add -host 192.168.2.100 dev lo:0
chmod +x /etc/rc.d/rc.local

(2) Adjust the kernel's ARP response parameters

vim /etc/sysctl.conf
......
net.ipv4.conf.lo.arp_ignore = 1 #系统只响应目的IP为本地IP的ARP请求
net.ipv4.conf.lo.arp_announce = 2 #系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

sysctl -p

yum install -y nfs-utils rpcbind httpd
systemctl start rpcbind
systemctl enable rpcbind
systemctl start httpd.service
systemctl enable httpd.service

(3) The first two steps of the two web servers are the same, but the following configuration is not the same

Web1 server: ens33: 192.168.2.6 lo:0 (VIP): 192.168.2.100

showmount -e 192.168.2.5

mount.nfs 192.168.2.5:/opt/test1 /var/www/html
df -h
echo 'this is test1 web!' > /var/www/html/index.html

Web2 server: ens33: 192.168.2.7 lo:0 (VIP): 192.168.2.100

showmount -e 192.168.2.5

mount.nfs 192.168.2.5:/opt/test2 /var/www/html
df -h
echo 'this is test2 web!' > /var/www/html/index.html

3. Configure the load scheduler

Load scheduler: 192.168.2.4 lo:0 (VIP): 192.168.2.100

(1) Turn off the firewall and load the ip_vs module

systemctl stop firewalld.service 
systemctl disable firewalld.service 
setenforce 0

#加载ip_vs模块，并安装ipvsadm工具
modprobe ip_vs
cat /proc/net/ip_vs
yum install -y ipvsadm

(2) Configure virtual IP address (VIP: 192.168.2.100)

vim /etc/sysconfig/network-scripts/ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.2.100
NETMASK=255.255.255.255

ifup ens33:0
ifconfig ens33:0

(3) Adjust proc response parameters

#由于 LVS 负载调度器和各节点需要共用 VIP 地址，应该关闭Linux 内核的重定向参数响应，不充当路由器，
vim /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0

sysctl -p

(4) Configure load distribution strategy

ipvsadm-save > /etc/sysconfig/ipvsadm
或者
ipvsadm --save > /etc/sysconfig/ipvsadm

systemctl start ipvsadm.service

#清除原有策略
ipvsadm -C
ipvsadm -A -t 192.168.2.100:80 -s rr
ipvsadm -a -t 192.168.2.100:80 -r 192.168.2.6:80 -g #如果这里是隧道模式，直接将-g替换成-i即可
ipvsadm -a -t 192.168.2.100:80 -r 192.168.2.7:80 -g

#查看节点状态，Route代表 DR模式
ipvsadm -ln

4. Test verification

Visit http://192.168.2.100/ on the client, refresh to test whether the load balance is successful