LVS load balancing cluster NAT mode

1. Clustering and Distribution

1.1. The meaning of cluster

• cluster, a cluster, a single system formed by combining multiple computers to solve a specific problem.
• It consists of multiple hosts, but only appears as a whole to the outside world.

1.2, lvs model

1.3. System performance expansion method

• scale UP: vertical expansion, upward expansion, enhancement, more powerful computers running the same service
• Scale Out: horizontal expansion, adding equipment, running multiple services in parallel, scheduling and allocation issues.
  The performance of a single computer has an upper limit, and it is impossible to expand vertically without limit. Multi-core CPUs mean that even a single computer can be parallelized. So why not parallelize the technology in the first place

1.4. Three types of clusters

• Load balancing cluster
• High availability cluster
• High performance computing cluster

1.4.1. Load balancing cluster

LB: Load balancing consists of multiple hosts, each host only bears part of the access requests

• Improve the response capability of the application system, handle as many access requests as possible, reduce latency targets, and obtain high concurrency and high-load overall performance
• The load distribution of LB depends on the offloading algorithm of the master node.

1.4.2. High availability cluster

HA; high availability, avoid SPOF

• Improve the reliability of application systems, minimize interruption time as the goal, ensure service continuity, and achieve high availability and fault tolerance.
• HA works in two modes: duplex and master-slave.

1.4.3. High-performance computing cluster

High-performance computing cluster
HPC: high performance

• The goal is to improve the CPU computing speed of the application system, expand hardware resources and analysis capabilities, and obtain high-performance computing capabilities equivalent to large-scale supercomputers.
• High performance relies on "distributed" computing and "parallel computing". Through dedicated hardware and software, the CPU, memory and other resources of multiple servers are integrated to achieve computing capabilities that only large-scale, supercomputers have.

1.5. LVS load scheduling algorithm

1.5.1. Polling

• Allocate received access requests to each node in the cluster in turn
• Treat each server equally, regardless of the actual number of connections and system load on the server

1.5.2. Weighted polling

• Distribute requests according to the weight value set by the scheduler
• Nodes with higher weights get tasks first and are assigned more requests.
• This ensures that nodes with high performance can bear more requests.

1.5.3. Minimum connections

• Allocation based on the number of established connections to the real server
• Prioritize received access requests to the node with the fewest connections
• If all server nodes have similar performance, this approach can provide better load balancing.

1.5.4. Weighted least connection

• When the performance of server nodes varies greatly, the scheduler can automatically adjust the weight according to the node server load.
• Nodes with higher weight will bear a greater proportion of the load of active connections

1.5.5、ip_hash

• Perform hash calculation based on the IP address of the request source to obtain the backend server
• In this way, requests from the same IP will always be processed by the same server, so that the request context information can be stored on this server.

1.5.6、url_hash

• Distribute requests according to the hash result of the accessed URL so that each URL is directed to the same backend server
• It is more effective when the backend server is cached

1.5.7、 fail

• The rotation balancing algorithm used by the built-in load balancing is not used.
• Instead, it can intelligently perform load balancing based on page size and loading time.
• That is to say, user requests are allocated according to the backend server time, and those with short response times are allocated first.

1.6. Distributed system

• Distributed storage: Store data dispersedly on multiple independent devices. Ceph, GlusterFS, FastDFS, MogileFS
• Distributed computing: Break the application into many small parts and assign them to multiple computers for processing. Hadoop,Spark
• Distributed common applications
  • Distributed applications - services are split according to functions and use microservices (a single application is divided into a group of small services, and the services coordinate and cooperate with each other to provide users with final value services)
  • Distributed static resources – static resources are placed on different storage clusters
  • Distributed data and storage – using a key-value caching system
  • Distributed computing – use distributed computing for special services, such as Hadoop clusters

1.7. Clustering and distribution

Distribution improves efficiency by shortening the execution time of a single task, while clustering improves efficiency by increasing the number of tasks executed per unit time.

• Cluster: The same business system is deployed on multiple servers. In the cluster, there is no difference in the functions implemented by each server, and the data and code are the same.

• Distributed: A business is split into multiple sub-businesses, or is itself a different business, deployed on multiple servers. In a distributed system, the functions implemented by each server are different, and the data and code are also different. The functions of each distributed server are added together to form a complete business.

1.8. Cluster design principles

1. Scalability—the horizontal expansion capability of the cluster
2. Availability—downtime (SLA service level agreement)
3. Performance—access response time
4. Capacity—maximum concurrent throughput per unit time (C10K concurrency problem)

1.9. Cluster design and implementation

1.9.1. Infrastructure level

• Improve hardware resource performance—Use higher-performance hardware resources from the ingress firewall to the back-end web server

• Multiple domain names—DNS polling A record resolution

• Multiple portals—parse A records to multiple public IP portals

• Multiple computer rooms—same city + remote disaster recovery

• CDN (Content Delivery Network)—achieves global load balancing based on GSLB (Global Server Load Balance), such as: DNS

1.9.2. Business level

• Layering: security layer, load layer, static layer, dynamic layer, (cache layer, storage layer) persistence and non-persistence

• Segmentation: Split large businesses into small services based on functions

• Distributed: For businesses in special scenarios, use distributed computing

1.10. Load balancing cluster architecture

1. The first layer, load scheduler (Load Balancer or Director)

2. The second layer, Server Pool (Server Pool)

3. The third layer, shared storage (Share Storage)

1.11, LB Cluster load balancing cluster

1.11.1. Divided by implementation method

• Hardware
  F5 Big-IP (F5 Server Load Balancing Module)
  
  Citrix Netscaler

A10 A10

• software

lvs	Linux Virtual Server, used by Alibaba four-layer SLB (Server Load Balance)
nginx	Supports seven-layer scheduling, Alibaba seven-layer SLB uses Tengine
haproxy	Supports seven-layer scheduling
ats	Apache Traffic Server, yahoo donates to apache
perbal	Written in Perl

1.11.2. Work-based protocol hierarchy division

• Transport layer (generic): DNAT and DPORT
  LVS:
  nginx: stream
  haproxy: mode tcp
• Application layer (dedicated): For specific protocols, often called proxy server
  http: nginx, httpd, haproxy(mode http), …
  fastcgi: nginx, httpd, …
  mysql: mysql-proxy, mycat…

1.11.3. Load balancing session persistence

1.session sticky: the same user schedules a fixed server
Source IP: LVS sh algorithm (for a specific service)
Cookie

2.session replication: Each server has all sessions (replication)
session multicast cluster

3.session server: dedicated session server (server)
Memcached, Redis

1.12. HA high availability cluster implementation

keepalived: vrrp protocol
Ais: application interface specification
heartbeat
cman+rgmanager (RHCS)
coresync_pacemaker

2. Introduction to Linux Virtual Server

2.1. Introduction to LVS

LVS: Linux Virtual Server, load scheduler, kernel integration, Zhang Wensong (Hua Ming Zhengming), Alibaba’s four-layer SLB (Server Load Balance) is based on LVS+keepalived

LVS 官网：http://www.linuxvirtualserver.org/
阿里SLB和LVS：
https://yq.aliyun.com/articles/1803
https://github.com/alibaba/LVS

The entire SLB system consists of three parts: four-layer load balancing, seven-layer load balancing and control system

Four-layer load balancing uses open source software LVS (linux virtual server) and is customized according to cloud computing needs; this technology has been fully implemented in Alibaba’s internal business for 2 years;

Seven-layer load balancing, using the open source software Tengine; this technology has been fully implemented in Alibaba’s internal business for more than 3 years; · Control system, used to configure and monitor the load balancing system;

2.2. Working principle of LVS

VS schedules and forwards the request message to a certain RS based on the target IP, target protocol, and port, and selects the RS based on the scheduling algorithm. LVS is a kernel-level function that works in the INPUT chain and "processes" the traffic sent to INPUT.

2.3. LVS functions and organizational structure

The application scenario of load balancing is for high-traffic services to improve the availability and reliability of applications.

2.3.1. Applied to businesses with high traffic volume

If your application has a high traffic volume, you can configure listening rules to distribute the traffic to different cloud server ECS (Elastic Compute Service) instances. In addition, the session persistence feature can be used to forward requests from the same client to the same backend ECS

2.3.2. Extended applications

ECS instances can be added and removed at any time according to business development needs to expand the service capabilities of the application system. It is suitable for various Web servers and App servers.

2.3.3. Eliminate single points of failure

Multiple ECS instances can be added under the load balancing instance. When some of the ECS instances fail, the load balancing will automatically block the failed ECS instances and distribute the requests to the normally running ECS ​​instances to ensure that the application system can still work normally.

2.4. Terminology in LVS cluster types

• VS: Virtual Server, Director Server (DS), Dispatcher (scheduler), Load Balancer (lvs server)

• RS: Real Server(lvs), upstream server(nginx), backend server(haproxy)(real server)

• CIP: Client IP (client IP)

• VIP: Virtual serve IP VS external network IP

• DIP: Director IP VS intranet IP

• RIP: Real server IP (real IP

3. LVS working mode and related commands

3.1. Working mode of LVS cluster

• Load balancing clusters are currently the most commonly used cluster type by enterprises.

• The cluster's load scheduling technology has three working modes:

  • lvs-nat: Modify the target IP of the request message, DNAT of multiple target IPs

  • lvs-dr: Manipulate encapsulation of new MAC address (direct routing)

  • lvs-tun: tunnel mode

3.1.1. NAT mode of LVS

lvs-nat: It is essentially DNAT of multiple target IPs . It implements forwarding by modifying the target address and target port in the request message to the RIP and PORT of the RS somewhere
(1) RIP and DIP should be in the same IP network, and A private network address should be used; the gateway of RS should point to DIP
(2) Both request messages and response messages must be forwarded through the Director, which can easily become a system bottleneck
(3) Port mapping is supported, and the target PORT of the request message can be modified
(4 ) VS must be a Linux system, RS can be any OS system

3.1.2, IP tunnel

1. RIP and DIP do not need to be in the same physical network. The gateway of RS generally cannot point to DIP, and RIP can communicate with the public network. In other words, cluster nodes can be implemented across the Internet. DIP, VIP, RIP can be public network addresses.

2. A VIP address needs to be configured on the channel interface of RealServer in order to receive the data packets forwarded by DIP and the source IP of the response packets.

3. When DIP is forwarded to RealServer, a tunnel is required. The source IP of the outer IP header of the tunnel is DIP and the target IP is RIP. The IP header that RealServer responds to the client is obtained based on the analysis of the IP header of the inner layer of the tunnel. The source IP is VIP and the destination IP is CIP

4. The request message must go through the Director, but the response does not go through the Director. The response is completed by the RealServer itself.

5. Does not support port mapping

6.RS’s OS must support tunnel function

3.1.3. Direct routing

• Direct Routing: DR mode for short, adopts a semi-open network structure, similar to the structure of TUN mode, but the nodes are not scattered everywhere, but are located on the same physical network as the scheduler.

• The load scheduler is connected to each node server through the local network. There is no need to establish a dedicated IP tunnel for direct routing. The LVS default mode is the most widely used. It forwards by re-encapsulating a MAC header in the request message. The source MAC is the interface where the DIP is located. MAC, the target MAC is the MAC address of the interface where the RIP of a selected RS is located;

• The source IP/PORT and destination IP/PORT remain unchanged.

3.1.4. Summary and comparison of LVS working modes

	NAT	AGAIN	DR
advantage	port conversion	WAN	best performance
Disadvantages Performance bottleneck	Server supports tunnel mode	Cross-network segments are not supported
Real server requirements	any	Tunneling	Non-arp device
Support network	private (private network)	LAN/WAN (private network/public network)	LAN (private network)
Number of real servers	low (10~20)	High (100)	High (100)
real server gateway	lvs intranet address	Own router (defined by network workers)	Own router (defined by network workers)

4. ipvsadm tool

4.1 ipvsadm tool option description

Options	explain
-A	Add virtual server
-D	Delete the entire virtual server
-s	Specify the load scheduling algorithm (round robin: rr, weighted round robin: wrr, least connections: lc, weighted least connections: wlc)
-a	Indicates adding a real server (node ​​server)
-d	Delete a node
-t	Specify VIP address and TCP port
-r	Specify RIP address and TCP port
-m	Indicates using NAT cluster mode
-g	Indicates using DR mode
-i	Indicates using TUN mode
-w	Set the weight (when the weight is 0, it means suspending the node)
-p 60	Indicates keeping a long connection for 60 seconds
-l	List to view LVS virtual servers (default is to view all)
-n	Display address, port and other information in numerical form, often used in combination with the "-" option. ipvsadm-ln

5. LVS load balancing deployment in NAT mode

Configuration Environment

负载调度器：配置双网卡 内网：192.168.11.14(ens33)  外网卡：12.0.0.1(ens36)
二台WEB服务器集群池：192.168.11.12 192.168.11.13 
一台NFS共享服务器：192.168.11.11
客户端：

5.1 Deploy shared storage (NFS server: 192.168.11.11)

• NFS is a network file system protocol based on TCP/IP transmission, originally developed by Sun. By using the NFS protocol, clients can access shared resources on a remote server as if they were local directories. For most load balancing clusters, it is a common practice to use the NFS protocol to share data storage. NFS is also a protocol that NAS storage devices must support.

• The implementation of NFS services relies on the RPC (Remote Process Call) mechanism to complete the remote-to-local mapping process. In CentOS 7 system, nfs-utils and rpcbind software packages need to be installed to provide NFS share services. The former is used for NFS share publishing and access, and the latter is used for RPC support.

1. #关闭防火墙
systemctl stop firewalld
setenforce 0
 
2. #安装nfs服务
yum install nfs-utils rpcbind -y
 
3. #新建目录，并创建站点文件
cd /opt/
mkdir yxp dhc
echo "this is yxp" >yxp/index.html
echo "this is dhc" >dhc/index.html
 
4. #开启服务
systemctl start rpcbind
systemctl start nfs
 
5. #授权
chmod 777 yxp/ dhc/
 
6. #设置共享策略
vim /etc/exports
/opt/yxp 192.168.11.0/24(rw,sync)
/opt/dhc 192.168.11.0/24(rw,sync)
 
 
7. #发布服务
systemctl restart nfs
exportfs -rv
 
 

Turn off the firewall,

install the nfs service
, create a new directory, and create the site file

Start service

Authorize

Set sharing policy


Publish service

5.2 Node Web Server

The first station (192.168.11.12)

1. #关闭防火墙
systemctl stop firewalld
setenforce 0
 
2. #安装httpd
yum install -y httpd
 
3. #查看nfs服务
showmount -e 192.168.11.11
 
4. #挂载站点
#法一：临时挂载
df
cat /var/www/html/index.html
mount 192.168.11.11:/opt/yxp /var/www/html/
#法二：永久挂载
vim /etc/fstab
192.168.11.11:/opt/yxp/  /var/www/html/        nfs     defaults,_netdev 0 0 
mount -a
 
 
5. #开启httpd服务
systemctl start httpd
 
6. #指定网关
vim /etc/sysconfig/network-scripts/ifcfg-ens33 
GATEWAY=192.168.11.11
#DNS1=8.8.8.8
 
7. #重启网络服务
systemctl restart network

Turn off the firewall.

Install httpd

to view the nfs service

. Mount the site temporarily

and permanently.

Start the httpd service

and specify the gateway

Restart network service

second station (192.168.11.13)

1. #关闭防火墙
systemctl stop firewalld
setenforce 0
 
2. #安装httpd
yum install -y httpd
 
3. #查看nfs服务
showmount -e 192.168.11.11
 
4. #挂载站点
#法一：临时挂载
mount 192.168.11.11:/opt/dhc /var/www/html/
df
cat /var/www/html/index.html
#法二：永久挂载
vim /etc/fstab
192.168.11.11:/opt/dhc/  /var/www/html/        nfs     defaults,_netdev 0 0 
mount -a
 
5. #开启httpd服务
systemctl start httpd
 
6. #指定网关
vim /etc/sysconfig/network-scripts/ifcfg-ens33 
GATEWAY=192.168.11.11
#DNS1=8.8.8.8
 
7. #重启网络服务
systemctl restart network
 

Turn off the firewall.

Install httpd

to view the nfs service.

Temporarily mount the site. Mount

it permanently.


Enable the httpd service.

Specify the gateway.

Restart network service

5.3 load scheduler
Intranet: 192.168.11.11(ens33) External network card: 12.0.0.1(ens36)

1. #关闭防火墙
systemctl stop firewalld
setenforce 0
 
2. #安装ipvsadm
yum install -y ipvsadm*
 
 
3. #添加一块网卡，我这里是ens36，配置网卡，重启网络
cd /etc/sysconfig/network-scripts/
cp ifcfg-ens33 ifcfg-ens36
vim ifcfg-ens36
去掉网关、UUID、修改IP
vim ifcfg-ens33
去掉网关
systemctl restart network
 
4. #打开路由转发功能
vim /etc/sysctl.conf
net.ipv4.ip_forward = 1 
sysctl -p
 
5. #防火墙做策略
#查看策略
iptables -nL -t nat
#清空策略
iptables -F
#添加策略
iptables -t nat -A POSTROUTING -s 192.168.11.0/24 -o ens37 -j SNAT --to 12.0.0.1
#查看策略
iptables -nL -t nat
 
6. #加载LVS内核模块
modprobe ip_vs
cat /proc/net/ip_vs
 
7. #开启ipvsadm服务
ipvsadm-save >/etc/sysconfig/ipvsadm
systemctl start ipvsadm.service
 
8. #清空策略
ipvsadm -C
 
9. #制定策略
#指定IP地址 外网的入口  -s rr  轮询
ipvsadm -A -t 12.0.0.1:80 -s rr
#先指定虚拟服务器再添加真实服务器地址，-r:真实服务器地址 -m指定nat模式
ipvsadm -a -t 12.0.0.1:80 -r 192.168.11.12:80 -m
ipvsadm -a -t 12.0.0.1:80 -r 192.168.11.13:80 -m
#开启服务
ipvsadm
 
10. 查看策略
ipvsadm -ln

Turn off firewall

Install ipvsadm

and add a network card. Here is ens36. Configure the network card.

Turn on routing and forwarding function

Create a firewall policy.

Enable the routing function and refresh.

Enable the ipvsadm service.

Clear the policy.

Develop a viewing policy.

Connectivity test.