HA Cluster -lvs

lvs HA Cluster

Technical Description:

LVS集群采用IP负载均衡技术和基于内容请求分发技术。
调度器具有很好的吞吐率，将请求均衡地转移到不同的服务器上执行，
且调度器自动屏蔽掉服务器的故障，从而将一组服务器构成一个高性能的、高可用的虚拟服务器。
整个服务器集群的结构对客户是透明的，而且无需修改客户端和服务器端的程序。
为此，在设计时需要考虑系统的透明性、可伸缩性、高可用性和易管理性

Cluster three-tier structure:

In general, LVS cluster three-tier structure, the main components are:

• Load balancer (load balancer), it is the entire cluster to the outside of the front-end machine, the customer is responsible for sending a request to perform a set of servers, and the customer service is considered from a single IP address (we can call it a virtual IP address) Up

• Server Pool (server pool), is a group of servers to perform real customer requests, the service performed there WEB, MAIL, FTP, and DNS

• Shared storage (shared storage), which provides a shared storage area for the server pool, it is easy to make the server pool have the same content, providing the same services

lvs term cluster types:

vs：Virtual Server, Director, Dispatcher, Balancer
rs：Real Server, upstream server, backend server
CIP：Client IP
VIP: Virtual serve IP
DIP: Director IP
RIP: Real server IP

请求过程：CIP <–> VIP <==> DIP <–> RIP

lvs cluster type:

lvs-nat：修改请求报文的目标IP，然后转发给被选中的RS

lvs-dr：在请求报文的首部重新封装新的MAC地址，然后转发给被选中的RS

lvs-tun：在原请求IP报文之外新加一个IP首部<源地址为CIP，目标地址为RIP>，然后转发给RS

lvs-fullnat：修改请求报文的源和目标IP<源地址修改为DIP,目标地址修改为RIP>，然后转发给RS

lvs-night:

Similar multiple destination IP DNAT, the request by the destination address and destination port in the packet forwarding is modified to achieve a RIP and PORT picked the RS

(1) RIP和DIP必须在同一个IP网络，且应该使用私网地址；RS的网关要指向DIP
(2) 请求报文和响应报文都必须经由Director转发；Director易于成为系统瓶颈
(3) 支持端口映射，可修改请求报文的目标PORT
(4) vs必须是Linux系统，rs可以是任意系统

lvs-dr：

Direct Routing, direct routing

Forwarded through request packet is re-encapsulated a MAC header, MAC source MAC interface DIP is located, the destination MAC is the MAC address of a selected location of the RIP interface RS

• Source IP / PORT, and target IP / PORT remain unchanged;

• Director and each configured to use the VIP had RS;

(1) VIP地址要配置在VS主机和所有的RS主机上
(2) 确保前端路由器将目标IP为VIP的请求报文发往Director，限制RS主机上关于VIP的广播和应答：
    (a) 在前端网关做静态绑定；
    (b) 在RS上使用arptables；
    (c) 在RS上修改内核参数以限制arp通告及应答级别；
        arp_announce
        arp_ignore
(3) RS的RIP可以使用私网地址，也可以是公网地址；RIP与DIP在同一IP网络；RIP的网关不能指向DIP，以确保响应报文不会经由Director
(4) RS跟Director要在同一个物理网络，VS和所有RS主机之间不能跨路由器
(5) 请求报文要经由Director，但响应不能经由Director，而是由RS直接发往Client
(6) 不支持端口映射

lvs-tun:

Forwarding mode: does not modify the header request packet IP (source IP address is CIP, destination IP as VIP), whereas in the former to the IP packet and then encapsulated header an IP (source IP is DIP, destination IP is RIP), the packet sent to pick out targets RS

(1) DIP, VIP, RIP都应该是公网地址
(2) RS的网关不能，也不可能指向DIP
(3) RS的主机上也要配置VIP地址
(3) 请求报文要经由Director，但响应不能经由Director
(4) 不支持端口映射
(5) RS的OS得支持隧道功能

lvs-fullnat：

By simultaneous modification request packet's source IP address and destination IP address for forwarding;

• CIP –> DIP
• VIP –> RIP

(1) VIP是公网地址，RIP和DIP是私网地址，可以跨路由器，且通常不在同一IP网络；因此，RIP的网关一般不会指向DIP；
(2) RS收到的请求报文源地址是DIP，因此，只需响应给DIP；但Director还要将其发往Client；
(3) 请求和响应报文都经由Director；
(4) 支持端口映射；
注意：此类型默认不支持  

ipvs scheduler：

Consider whether the current load status of each RS according to its schedule, can be divided into two kinds of static method and dynamic method:

Static method: only scheduling based on the algorithm itself

RR：roundrobin，轮询；把所有的RS主机统计起来，每次有新请求来，以轮询的方式一个个的调度用户请求到RS主机上去

WRR：Weighted RR，加权轮询；把所有的RS主机统计起来，并记录每个RS主机的权值<在这里表示自身的某个服务的处理能力，能力越强，权值越高>，每次有新请求来，以权值的大小进行轮询，调度用户请求到RS主机上去，权值高的主机，会接收到更多的用户请求

SH：Source Hashing，实现session sticy，源IP地址hash；将来自于同一个IP地址的请求始终发往第一次挑中的RS，从而实现会话绑定

DH：Destination Hashing；目标地址哈希，将发往同一个目标地址的请求始终转发至第一次挑中的RS
    DH：提高VS后缓存服务器中缓存的命中率
    DH：多数情况用于缓存服务器

Dynamic method: mainly performs scheduling according to the current load status of each RS and a scheduling algorithm;

Overhead: VS scheduling comparator to a value smaller Overhead priority scheduling server

LC：least connections  最少连接
    Overhead=activeconns*256+inactiveconns
    基于上面的Overhead算法，计算所有RS主机的Overhead值，把新请求调度到Overhead值小的服务器

WLC：Weighted LC   加权的最少连接
    Overhead=(activeconns*256+inactiveconns)/weight
    基于上面的Overhead算法，计算所有RS主机的Overhead值，把新请求调度到Overhead值小的服务器

SED：Shortest Expection Delay
    Overhead=(activeconns+1)*256/weight
    基于上面的Overhead算法，计算所有RS主机的Overhead值，把新请求调度到Overhead值小的服务器

NQ：Never Queue
    永不排队，每个后端服务器，至少先分一个，然后在使用SED调度算法对新请求进行调度

LBLC：Locality-Based LC，动态的DH算法
    简单的理解就是：类似静态方法的DH算法，但它是动态的

LBLCR：LBLC with Replication，带复制功能的LBLC
    在LBLC上，后端服务器，基于session replication机制，复制一台RS主机的缓存到其他RS主机上，则当老用户进行请求的时候，不仅可以转发给老用户第一次访问的主机，也可以转发给RS其他主机，因为此时其他RS主机也有此老用户访问的缓存

lvs-nat configuration:

Topology:

lvs-nat data flow graph:

Design Points:

(1) RIP与DIP在同一IP网络, RIP的网关要指向DIP；
    VIP=192.168.0.100
    DIP=172.16.0.1
    RIP-1=172.16.0.2
    RIP-2=172.16.0.3

RS1:

yum -y install httpd net-tools
    
vim /var/www/html/index.html
<h1>RS1</h1>
    
ifconfig ens33:0 172.16.0.2/24 up
route add default gw 172.16.0.1
systemctl start httpd

RS2：

yum -y install httpd net-tools

vim /var/www/html/index.html
<h1>RS2</h1>
    
ifconfig ens33:0 172.16.0.3/24 up
route add default gw 172.16.0.1
systemctl start httpd

VS:

yum install ipvsadm net-tools
ifconfig ens33:0 172.16.0.1/24 up
echo 1 > /proc/sys/net/ipv4/ip_forward
ipvsadm -A -t 192.168.0.100:80 -s rr
ipvsadm -a -t 192.168.0.100:80 -r 172.16.0.2 -m -w 1
ipvsadm -a -t 192.168.0.100:80 -r 172.16.0.3 -m -w 1

lvs-dr configuration:

Topology:

LVS-DR analog data flow graph

As shown below: When a client requests a VIP, the request will be sent to Director (scheduler), the scheduler discovery request is a set of cluster services, according to a scheduling algorithm to forward the request to the RealServer, attention in the process of forwarding merely modified the MAC address of the data packets, so this is why we ask DR and RS must be in the same physical network, it is to ensure that data packets can be forwarded by modifying the MAC address.

In addition to configure each server on the network, the first of each service has a separate card required. In the DIP configuration DR on eth0, VIP will alias in the network configuration, this is to do after the scheduler easy availability VIP IP addresses do drift. RS needs the same in each of the VIP configured, so that a Client RS can respond directly, without the need to go through DR, but the physical network of a plurality of the same name is allowed IP, it is necessary to modify the kernel parameters RS, and VIP configured on LO, allowing it to remain VIP but does not allow outside broadcast, so packets sent from the RS the source address is the VIP, of course, need to go out from the real physical card, it is necessary to add a route, allowing access VIP request packet, in response thereto is arranged a VIP card, so that packets sent in response to the source address is the VIP, may be accepted Client.

Design Points:

VS上的VIP地址可以直接配置在DIP地址所在的网卡上，Linux主机上，一块网卡可以配置多个地址
RS上的VIP地址必须配置在lo接口上，并且还要关闭arp的响应和通告功能

外网接口：10.1.43.101
内网接口：172.16.0.1

VIP：172.16.0.10
DIP：172.16.0.9
RIP-1：172.16.0.2
RIP-2：172.16.0.3

dr model, VIP needs to be configured on each host, there are ways to resolve address conflicts of three steps:

(1) 在前端网关做静态绑定
(2) 在各RS使用arptables
(3) 在各RS修改内核参数，来限制arp响应和通告的级别
    限制响应级别：arp_ignore
        0：默认值，表示可使用本地任意接口上配置的任意地址进行响应
        1: 仅在请求的目标IP配置在本地主机的接收到请求报文接口上时，才给予响应
    
    限制通告级别：arp_announce
        0：默认值，把本机上的所有接口的所有信息向每个接口上的网络进行通告
        1：尽量避免向非直接连接网络进行通告
        2：必须避免向非本网络通告

RS pre-configuration script: <script to run on the host rs>

#!/bin/bash
#
vip=172.16.0.10
mask='255.255.255.255'
    
case $1 in
start)
  echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
  echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
  echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
  echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    
  ifconfig lo:0 $vip netmask $mask broadcast $vip up
  route add -host $vip dev lo:0
  route add default gw 172.16.0.1
    ;;
        
stop)
    ifconfig lo:0 down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ;;
        
*)
    echo "Usage $(basename $0) start|stop"
    exit 1
    ;;
esac

VS configuration script: <script can be run on the host vs>

#!/bin/bash
#
vip='172.16.0.10'
iface='eno16777736:0'
mask='255.255.255.255'
port='80'
rs1='172.16.0.2'
rs2='172.16.0.3'
scheduler='wrr'
type='-g'
    
case $1 in
start)
    ifconfig $iface $vip netmask $mask broadcast $vip up
    iptables -F
    ipvsadm -A -t ${vip}:${port} -s $scheduler
    ipvsadm -a -t ${vip}:${port} -r ${rs1} $type -w 1
    ipvsadm -a -t ${vip}:${port} -r ${rs2} $type -w 1
    ;;
        
stop)
    ipvsadm -C
    ifconfig $iface down
    ;;
        
*)
    echo "Usage $(basename $0) start|stop"
    exit 1
    ;;
esac

The router configuration:

ifconfig eno16777736 10.1.43.101/16 up
ifconfig eno33554984 10.1.43.101/16 up
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -d 10.1.43.101 -p tcp --dport 80 -j DNAT --to-destination 172.16.1.10:80

postscript

lvs-dr model: experimental design and configuration vip environment and not on the same network segment dip / rip achieve

Tip: Configure an alias IP network IP interfaces on the router, this IP in the same network segment with VIP

Design Points:

VS上的VIP地址可以直接配置在DIP地址所在的网卡上，Linux主机上，一块网卡可以配置多个地址

RS上的VIP地址必须配置在lo接口上，并且还要关闭arp的响应和通告功能

外网接口：10.1.43.101
内网接口：172.16.0.1 172.16.10.1

VIP：172.16.1.10
DIP：172.16.0.9
RIP-1：172.16.0.2
RIP-2：172.16.0.3

surroundings

主机：Centos6

Host computer	information
router	Centos6
Director	Centos6
RS1	Centos6
RS2	Centos6

Configuration on the router:

[root@node4 ~]# ifconfig eno16777736 10.1.43.101/16 up 
[root@node4 ~]# ifconfig eno33554984 172.16.0.1/24 up
[root@node4 ~]# ifconfig eno33554984:0 172.16.1.1/24 up       //配置一个别名，用于同VS主机上的VIP进行通信 

[root@node4 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward 
[root@node4 ~]# iptables -t nat -A PREROUTING -d 10.1.43.101 -p tcp --dport 80 -j DNAT --to-destination 172.16.1.10:80

On RS1 configuration:

[root@node1 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@node1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@node1 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@node1 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    
[root@node1 ~]# ifconfig lo:0 172.16.1.10 netmask 255.255.255.255 broadcast 172.16.1.10 up
    
[root@node1 ~]# yum -y install httpd
[root@node1 ~]# service httpd start
    
[root@node1 ~]# cd /var/www/html
[root@node1 www]# vim index.html
<h1>172.16.0.2</h1>

On RS2 configuration:

[root@node2 ~]# echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
[root@node2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@node2 ~]# echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
[root@node2 ~]# echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    
[root@node2 ~]# ifconfig lo:0 172.16.1.10 netmask 255.255.255.255 broadcast 172.16.1.10 up
    
[root@node2 ~]# yum -y install httpd
[root@node2 ~]# service httpd start
    
[root@node2 ~]# cd /var/www/html
[root@node2 www]# vim index.html
<h1>172.16.0.3</h1>

VS configuration:

[root@node3 ~]# ifconfig eth0:0 172.16.1.10 netmask 255.255.255.255 broadcast 172.16.1.10 up
    
[root@node3 ~]# ipvsadm -C
[root@node3 ~]# ipvsadm -A -t 172.16.1.10:80 -s rr
[root@node3 ~]# ipvsadm -a -t 172.16.1.10:80 -r 172.16.0.2 -g
[root@node3 ~]# ipvsadm -a -t 172.16.1.10:80 -r 172.16.0.3 -g

Reference Documents

https://blog.csdn.net/brad_chen/article/details/47807281