DR mode deployment of LVS

1. LVS-DR data packet flow analysis

In order to facilitate the principle analysis, the Client and the cluster machine are placed on the same network, and the route of the data packet is 1-2-3-4.

1. The client sends a request to the target VIP, and the Director (load balancer) receives it. At this time, the source MAC address is the Client MAC address, and the destination MAC address is the MAC address of the scheduler Director.

2. The Director selects RealServer_1 according to the load balancing algorithm, does not modify or encapsulate the IP message, but changes the MAC address of the data frame to the MAC address of RealServer_1, and then sends it on the LAN. At this time, the source MAC address is the MAC address of Director, and the destination MAC address is the MAC address of RealServer_1.

3. RealServer_1 receives this frame and finds that the target IP matches the local machine after decapsulation (RealServer is bound to VIP in advance), so it processes this message. Then re-encapsulate the message, and send the response message to the physical network card through the lo interface and then send it out. At this time, the source MAC address is the MAC address of RealServer_1, and the destination MAC address is the MAC address of the client.

4. The client will receive the reply message. The Client thinks that it gets the normal service, but does not know which server handles it.
Note: If it crosses the network segment, the message will be returned to the user via the router via the Internet

Second, the ARP problem in LVS-DR

1. In the LVS-DR load balancing cluster, both the load balancing and the node server must be configured with the same VIP address.
2. Having the same IP address in the local area network will inevitably cause disorder of ARP communication between servers.

• When the ARP broadcast is sent to the LVS-DR cluster, because the load balancer and the node server are connected to the same network, they will both receive the ARP broadcast.
• Only the front-end load balancer responds, and other node servers should not respond to ARP broadcasts.
  3. Process the node server so that it does not respond to ARP requests for VIP
• Use virtual interface lo:0 to carry VIP addresses
• Set the kernel parameter arp_ignore=1: the system only responds to ARP requests whose destination IP is the local IP.
  4. RealServer returns packets (source IP is VIP) that are forwarded by the router. When re-encapsulating the packets, the MAC address of the router must be obtained first.
  5. When sending an ARP request, Linux defaults to use the source IP address of the IP packet (ie VIP) as the source IP address in the ARP request packet, instead of the IP address of the sending interface, such as: ens33
  6. After the router receives the ARP request , Will update the ARP entry
  7. The original VIP corresponding to the Director's MAC address will be updated to the VIP corresponding to the MAC address of the RealServer
  8. The router will forward the new request message to the RealServer according to the ARP entry, resulting in the Director’s VIP failure
  solution:
• Process the node server and set the kernel parameter arp_announce=2: The system does not use the source address of the IP packet to set the source address of the ARP request, but selects the IP address of the sending interface.
  The setting method to solve the two problems of ARP
  Modify the /etc/sysctl.conf file

net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2

3. DR mode of LVS load balancing cluster

1. Data packet flow analysis

(1) The client sends a request to the Director Server (load balancer), and the requested data message (source IP is CIP, destination IP is VIP) reaches the kernel space.
(2) Director Server and Real Server are in the same network, and data is transmitted through the second-layer data link layer.
(3) The kernel space judges that the target IP of the data packet is the local VIP. At this time, IPVS (IP virtual server) compares whether the service requested by the data packet is a cluster service, and repackages the data packet if it is a cluster service. Modify the source MAC address to the MAC address of the Director Server and modify the destination MAC address to the MAC address of the Real Server. The source and destination IP addresses have not changed, and then send the data packet to the Real Server.
(4) The MAC address of the request message arriving at the Real Server is its own MAC address, and the message is received. The data packet re-encapsulates the message (the source IP address is VIP and the destination IP is CIP), and the response message is transmitted to the physical network card through the lo interface and then sent out.
(5) Real Server directly transmits the response message to the client.

2. Features of DR mode

(1) Director Server and Real Server must be in the same physical network.
(2) Real Server can use private address or public network address. If you use a public network address, you can directly access RIP through the Internet.
(3) Director Server serves as the access entrance of the cluster, but not as a gateway.
(4) All request messages go through Director Server, but reply response messages cannot go through Director Server.
(5) The gateway of the Real Server is not allowed to point to the Director Server IP, that is, the data packets sent by the Real Server are not allowed to pass through the Director Server.
(6) The lo interface on the Real Server is configured with the VIP IP address.

Four, simple small experiment LVS-DR mode

########DR模式LVS负载均衡群集部署#########
DR 服务器：192.168.241.3
Web 服务器1：192.168.241.4
Web 服务器2：192.168.241.5
vip :192.168.241.200
客户端：192.168.241.6

1、配置负载调度器（192.168.241.3）
systemctl stop firewalld.service
setenforce 0
modprobe ip_vs
cat /proc/net/ip_vs
yum -y install ipvsadm

(1)配置虚拟IP地址（VIP：192.168.241.200）
cd /etc/sysconfig/network-scripts
#若隧道模式，复制为ifcfg-tunl0
vim ifcfg-ens33:0
DEVICE=ens33:0
ONBOOT=yes
IPADDR=192.168.241.200
NETMASK=255.255.255.255
ifup ens33:0
ifconfig ens33:0
(2)调整proc响应参数
#由于LVS负载调度器和各节点需要共用VIP地址，应该关闭Linux内核的重定向参数响应，不充当路由器
vim /etc/sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.ens33.send_redirects = 0
sysctl -p
(3)配置负载分配策略
ipvsadm-save > /etc/sysconfig/ipvsadm
systemctl start ipvsadm
ipvsadm -C
ipvsadm -A -t 192.168.241.200:80 -s rr
ipvsadm -a -t 192.168.241.200:80 -r 192.168.241.4:80 -g		#若隧道模式，-g替换为-i
ipvsadm -a -t 192.168.241.200:80 -r 192.168.241.5:80 -g
ipvsadm
ipvsadm -ln				#查看节点状态，Router代表DR模式

2、部署共享存储（NFS服务器：192.168.241.6）
systemctl stop firewalld.service
setenforce 0

yum -y install nfs-utils rpcbind
mkdir /opt/kgc /opt/benet
chmod 777 /opt/kgc /opt/benet

vim /etc/exports
/opt/kgc 192.168.241.0/24(rw,sync,no_root_squash)
/opt/benet 192.168.241.0/24(rw,sync,no_root_squash)
echo 'this is benet web' > /opt/benet/index.html
echo 'this is kgc web' > /opt/kgc/index.html

systemctl start nfs
systemctl start rpcbind

3、配置节点服务器（192.168.241.4、192.168.241.5）
systemctl stop firewalld.service
setenforce 0
（1）配置虚拟IP地址（VIP：192.168.241.200）
#此地址仅用作发送Web响应数据包的源地址，并不需要监听客户机的访问请求（改由调度器监听并分发）。因此使用需接口lo:0来承载VIP地址，并为本机添加一条路由记录，将访问VIP的数据限制在本地，以避免通信紊乱
cd /etc/sysconfig/network-scripts		##需要将ens33的网关和DNS注释
cp ifcfg-lo ifcfg-lo:0
vim ifcfg-lo:0
DEVICE=lo:0
ONBOOT=yes
IPADDR=192.168.241.200
NETMASK=255.255.255.255
ifup lo:0
ifconfig lo:0

route add -host 192.168.241.200 dev lo:0
或者vim /etc/rc.local
/sbin/route add -host 192.168.241.200 dev lo:0
chmod +x /etc/rc.d/rc.local

(2)调整内核的ARP响应参数以阻止更新VIP的MAC地址，避免发生冲突
vim /etc/sysctl.conf
......
net.ipv4.conf.lo.arp_ignore = 1				#系统只响应目的IP为本地IP的ARP请求
net.ipv4.conf.lo.arp_announce = 2			#系统不使用IP包的源地址来设置ARP请求的源地址，而选择发送接口的IP地址
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
sysctl -p
或者
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce

yum -y install nfs-utils rpcbind httpd
systemctl start rpcbind
systemctl start httpd

####192.168.241.4
mount 192.168.241.6:/opt/kgc /var/www/html


####192.168.241.5
mount 192.168.241.6:/opt/benet /var/www/html

4. Test the LVS cluster
. Use a browser on the client to visit http://192.168.241.200, and the default gateway points to 192.168.241.200