Nginx builds LVS-DR mode

1 Environmental preparation

1.1 Server and IP planning

LVS server
VIP (Virtual IP): 192.168.51.103
DIP (forwarder IP/intranet IP): 192.168.51.4

Two Nginx servers (RealServer)
RIP (Real IP/Intranet IP): 192.168.51.5
RIP (real IP/intranet IP): 192.168.51.6

1.2 Stop the NetworkManager service

The three servers execute the following two lines of commands separately, because the server uses a virtual machine and needs to be stopped NetworkManager, otherwise there will be problems

[root@localhost network-scripts]# systemctl stop NetworkManager
[root@localhost network-scripts]# systemctl disable NetworkManager

2 Configure LVS nodes and ipvsadm

2.1 Create subinterface

LVS服务器

[root@localhost network-scripts]# cd /etc/sysconfig/network-scripts
[root@localhost network-scripts]# ls
ifcfg-ens33      ifcfg-lo     ifdown-eth   ifdown-isdn  ifdown-routes  ifdown-TeamPort  ifup-aliases  ifup-ippp  ifup-plip   ifup-ppp     ifup-Team      ifup-wireless      network-functions-ipv6
ifdown       ifdown-ippp  ifdown-post  ifdown-sit     ifdown-tunnel    ifup-bnep     ifup-ipv6  ifup-plusb  ifup-routes  ifup-TeamPort  init.ipv6-global
ifcfg-ens33.bak  ifdown-bnep  ifdown-ipv6  ifdown-ppp   ifdown-Team    ifup             ifup-eth      ifup-isdn  ifup-post   ifup-sit     ifup-tunnel    network-functions
[root@localhost network-scripts]# cp ifcfg-ens33 ifcfg-ens33:1

2.2 Modify the configuration file

[root@localhost network-scripts]# vi ifcfg-ens33:1

BOOTPROTO=static
DEVICE=ens33:1
ONBOOT=yes
IPADDR=192.168.51.103
NETMASK=255.255.255.0

2.3 Refresh configuration

[root@localhost network-scripts]# service network restart

2.4 Install ipvsadm

• It seems that Alibaba Cloud does not support virtual IP, buy the load balance of the official website
• It seems that Tencent Cloud supports a maximum number of virtual IPs of 10

ipvsadm -Ln

[root@localhost network-scripts]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn

3 Configure virtual IP for two RSs

The configuration of the two RS machines is the same, and the following operations need to be performed 两台机器上都进行操作.

3.1 Copy configuration

[root@localhost network-scripts]# cd /etc/sysconfig/network-scripts
[root@localhost network-scripts]# ls
ifcfg-ens33       ifdown-eth   ifdown-isdn  ifdown-routes  ifdown-TeamPort  ifup-aliases  ifup-ippp  ifup-plip   ifup-ppp     ifup-Team      ifup-wireless      network-functions-ipv6
ifcfg-ens33.bak  ifdown       ifdown-ippp  ifdown-post  ifdown-sit     ifdown-tunnel    ifup-bnep     ifup-ipv6  ifup-plusb  ifup-routes  ifup-TeamPort  init.ipv6-global
ifcfg-lo         ifdown-bnep  ifdown-ipv6  ifdown-ppp   ifdown-Team    ifup             ifup-eth      ifup-isdn  ifup-post   ifup-sit     ifup-tunnel    network-functions
[root@localhost network-scripts]# cp ifcfg-lo ifcfg-lo:1

3.2 Modify configuration

[root@localhost network-scripts]# vi ifcfg-lo:1

DEVICE=lo:1
IPADDR=192.168.51.103
NETMASK=255.255.255.255
NETWORK=127.0.0.0
# If you're having problems with gated making 127.0.0.0/8 a martian,
# you can change this to something else (255.255.255.255, for example)
BROADCAST=127.255.255.255
ONBOOT=yes
NAME=loopback

3.3 Refresh configuration

方式一

[root@localhost network-scripts]# ifup lo

方式二

[root@localhost network-scripts]# service network restart

4 Configure arp for two RSs

4.1 Introduction to arp

ARP response level and notification behavior.

1.arp-ignore: ARP response level (processing request)

• 0: As long as the machine is configured with ip, it can respond to requests
• 1: The requested target address reaches the corresponding network interface before responding to the request

2. arp-announce: ARP announcement behavior (return response)

• 0: Any network interface on the machine is notified to the outside, and all network cards can receive the notification
• 1: As far as possible, avoid this network card and the target that does not match the announcement
• 2: Only in the cartoon report on this website

4.2 Modify the configuration file

[root@localhost network-scripts]# vi /etc/sysctl.conf

# configration for lvs
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

4.3 Refresh configuration

[root@localhost network-scripts]# sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2

4.4 Add gateway

[root@localhost network-scripts]# route add -host 192.168.51.103 dev lo:1

Set up开机添加

[root@localhost network-scripts]# echo "route add -host 192.168.51.103 dev lo:1" >> /etc/rc.local

5 Use ipvsadm to configure cluster rules

5.1 Create LVS Node

Cluster scheduler for users to access data

[root@localhost network-scripts]# ipvsadm -A -t 192.168.51.103:80 -s rr -p 5

• -A: Add cluster
• -t: tcp protocol
• ip address: set the access ip of the cluster, which is the virtual ip of LVS
• -s: set the algorithm of load balancing, rr means polling
• -p: Set the connection persistence time

5.2 Create 2 RS real servers

[root@localhost network-scripts]# ipvsadm -a -t 192.168.51.103:80 -r 192.168.51.5:80 -g
[root@localhost network-scripts]# ipvsadm -a -t 192.168.51.103:80 -r 192.168.51.6:80 -g

5.3 Save to rule base

Do not save, restart will be invalid

[root@localhost network-scripts]# ipvsadm -S

5.4 Check the cluster

查看集群列表

[root@localhost network-scripts]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.51.103:80 rr persistent 5
  -> 192.168.51.5:80              Route   1      0          0         
  -> 192.168.51.6:80              Route   1      0          0    

查看集群状态

[root@localhost network-scripts]# ipvsadm -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  192.168.51.103:80                   0        0        0        0        0
  -> 192.168.51.5:80                     0        0        0        0        0
  -> 192.168.51.6:80                     0        0        0        0        0

5.5 Other commands

# 重启ipvsadm，重启后需要重新配置
service ipvsadm restart
# 查看持久化连接
ipvsadm -Ln --persistent-conn
# 查看连接请求过期时间以及请求源ip和目标ip
ipvsadm -Lnc
# 设置tcp tcpfin udp 的过期时间（一般保持默认）
ipvsadm --set 1 1 1
# 查看过期时间
ipvsadm -Ln --timeout

详细的帮助文档

ipvsadm -h
man ipvsadm

6 Related information

• The blog post is not easy, everyone who has worked so hard to pay attention and praise, thank you