LVS basis

LVS basis

A, lvs basis

1. lvs concept

effect

Linux Virtual Server, load balancer, integrated core functions, mainly to address the high concurrency and load balancing issues, working in four layers, usually at the entrance as user access (behind the firewall), Ali's four SLB (Server Load Balance) is based on the realization LVS + keepalived

Access to the process:

CIP <–> VIP == DIP <–> RIP

Lvs-related terms:

• VS：Virtual Server，Director Server(DS)

• The Dispatcher (scheduler), Load Balancer

• RS：Real Server(lvs), upstream server(nginx)

• backend server(haproxy)

• CIP：Client IP

• VIP: Virtual serve IP VS external network IP

• DIP: Director IP VS within the IP network

• RIP: Real server IP

2. lvs principle

Normal data packet flow, such as blue, 3, and 4 as lvs layer balanced servers, and no ability to respond to customer requests directly, but forwards the request from the user to the back-end server, the packet entering to the kernel space and user space does not flow to go, but after lvs to modify the packet sent out

3. lvs cluster type

• lvs-nat: modification request packet destination IP, destination IP multi DNAT

  The disadvantage is that the request packets and response packets have been vs, vs too much pressure, and there is a single point of failure problem

• lvs-dr: new MAC address manipulation package

  Only response packets pass through vs, the response message sent by the rs. But can not cross the route between vs and rs

• lvs-tun: in addition to the original request IP packets plus a new IP header

  You can solve the routing problem across, but is not commonly used

• lvs-fullnat: modification request packet source and destination IP

  Almost no, there is no particular advantage

All the models can not solve lvs birth defects, can not be solved rs checking back-end server

4. lvs cluster tools

• ipvsadm: user space command-line tool, Rules Manager is used to manage the cluster service and RealServer

  == Note: == lvs rule is stored in the kernel, ipvsadm only management tools, regardless of whether the installation will not affect regular role

  root@z2:~# apt install ipvsadm 
  

• ipvs: kernel space frame work on the netfilter hook INPUT

  View linux kernel support for the lvs

  root@z1:~# grep -i -A 10 "ipvs" /boot/config-4.15.0-54-generic 
  

5. lvs scheduling algorithm

• Static algorithm

  1. RR: roundrobin, poll
  2. WRR: Weighted RR, WRR
  3. SH: Source Hashing, implement session sticky, the source IP address of the hash; to a request from the same IP address is always sent to the first pick of the RS, bound to the session in order to achieve
  4. DH: Destination Hashing; destination address hash, the first round robin scheduling to the RS, the subsequent request will be sent to the same destination address are always forwarded to the first pick of the RS, a typical usage scenario is forward proxy cache in the scene load balancing, such as: broadband operators

• Dynamic method: The main scheduling Overhead = value smaller RS ​​will be scheduled according to the current load status of each RS and Scheduling Algorithm

  1. LC: least connections
     for long connected applications Overhead = activeconns * 256 + inactiveconns (who is small, pick who)

     Minimum connection scheduling algorithm is to assign a new connection request to the server with the smallest number of currently connected, the minimum connection scheduling algorithm is a dynamic scheduling short, it is connected to the server by the number of currently active server load balancing is estimated, the scheduler needs recording each server has established a number of connections, when a request is dispatched to a server that connections plus 1, when the right connection abort or a timeout, which connections minus one, when the system implementation, we also introduce when the value of the server is 0, it indicates that the server can not be used without scheduled. This algorithm ignores server performance issues, some server performance, some poor server performance, distinguished performance by heavy weights, so with the following algorithm wlc

  2. WLC:Weighted LC
     The default scheduling method Overhead = (activeconns * 256 + inactiveconns) / weight (who is small, pick who) it is because the use of LC algorithm can not consider the old and new server performance

  3. SED: Shortest Expection Delay
     initial connection weights higher priority Overhead = (activeconns + 1) * 256 / weight

     Irrespective of inactive connections, whose weight is greater, we prefer big right server to receive the request, but the question will arise, when the load is initially empty, the weight of the larger servers will be very busy, all requests will go hair right above a major, but relatively small weight of the server is very busy, even not receive the request, so there was the following algorithm nq.

  4. NQ: Never Queue
     first round evenly distributed, subsequent SED

  5. LBLC: Locality-Based LC
     dynamic DH algorithm, be used: Forward proxy implemented in accordance with the load state

     Locality-Based Least Connections load balancing algorithm for scheduling request packet destination IP address, mainly for Cache cluster system, because Cache cluster client request packet destination IP address is changed, assuming any back-end the server can handle any request, the algorithm designed at substantially balanced load on the server, the scheduling request the same destination IP address to the same server, and to improve the locality of the primary server memory Cache hit ratio, to adjust the processing capacity of the entire cluster system.

  6. LBLCR: LBLC with Replication
     with LBLC copy functions, solve problems LBLC unbalanced load, copied from a heavy load to a light load of RS

Two, ipvsadm common usage

installation

root@z2:~# apt install ipvsadm 

The resulting file

root@z2:~# ll /sbin/ipvsadm
-rwxr-xr-x 1 root root 68912 May 24  2019 /sbin/ipvsadm*

root@z2:~# ll /sbin/ipvsadm*
-rwxr-xr-x 1 root root 68912 May 24  2019 /sbin/ipvsadm*
-rwxr-xr-x 1 root root   573 Feb  9  2015 /sbin/ipvsadm-restore*
-rwxr-xr-x 1 root root   744 Feb  9  2015 /sbin/ipvsadm-save*

The main program: / sbin / ipvsadm
rules save tool: / sbin / ipvsadm-save
rule reload tool: / sbin / ipvsadm-restore
the configuration file: /etc/ipvsadm.rules

ipvsadm command

Core functions:
Cluster service management: add, delete, change
RS cluster service management: add, delete, change, view

Command

• ipvsadm -A | E virtual-service [ -s scheduler] [-p [timeout]] [-M netmask] [-pe persistence_engine] [-b sched-flags]
  to add a new service address vs listening. Note that a server can add multiple

• ipvsadm -D virtual-service

  To delete a record vs Service

• ipvsadm -C

  Clear all records vs kernel table.

• ipvsadm -R

  Restore virtual server rules

• ipvsadm -S [-n]

  Save the virtual server rules, it outputs a readable format options -R

• ipvsadm -a|e virtual-service -r server-address [options]

  Add real back-end server address is listening vs

• ipvsadm -d virtual-service -r server-address

  Delete back-end server address is listening vs

• ipvsadm -L|l [virtual-service] [options]

  Displays a list of all vs

• ipvsadm -Z [virtual-service]

  # Virtual table service counter is cleared (clear the current number of connections, etc.)

• ipvsadm --set tcp tcpfin udp

  # Set the timeout value

• ipvsadm --start-daemon state [–mcast-interface interface] [–syncid sid]

  Start the synchronization daemon. He may be a master or behind backup, it is used to illustrate the LVS Router master or backup. On the VRRP function keepalived this function can also be used.

• ipvsadm --stop-daemon state

  Stop Sync Daemon

options

-t --tcp-service service-address #说明虚拟服务器提供的是tcp 的服务[vip:port] or [real-server-ip:port]
-u --udp-service service-address #说明虚拟服务器提供的是udp 的服务[vip:port] or [real-server-ip:port]
-f --fwmark-service fwmark #说明是经过iptables 标记过的服务类型。
-s --scheduler scheduler #使用的调度算法，有这样几个选项rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq,默认的调度算法是： wlc.
-p --persistent [timeout] #持久稳固的服务。这个选项的意思是来自同一个客户的多次请求，将被同一台真实的服务器处理。timeout 的默认值为300 秒。
-M --netmask #子网掩码
-r --real-server server-address #真实的服务器[Real-Server:port]
-g --gatewaying 指定LVS 的工作模式为直接路由模式（也是LVS 默认的模式）
-i --ipip #指定LVS 的工作模式为隧道模式
-m --masquerading #指定LVS 的工作模式为NAT 模式
-w --weight weight #真实服务器的权值
--mcast-interface interface #指定组播的同步接口
-c --connection #显示LVS 目前的连接 如：ipvsadm -L -c
--timeout #显示tcp tcpfin udp 的timeout 值 如：ipvsadm -L --timeout
--daemon #显示同步守护进程状态
--stats #显示统计信息
--rate #显示速率信息
--sort #对虚拟服务器和真实服务器排序输出
--numeric -n #输出IP 地址和端口的数字形式

Explain ipvs output items

1. ipvsadm -Ln

root@director:~# ipvsadm  -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.0.0.100:80 wrr
  -> 192.168.1.104:80             Route   1      0          5         
  -> 192.168.1.105:80             Route   1      0          6    

• Forward-forward, forward current is routed
• Weight Weight
• ActiveConn currently active connections
• InActConn not currently active connections

2. ipvsadm -Ln --stats

root@director:~# ipvsadm  -Ln --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
TCP  10.0.0.100:80                      27      171        0    11140        0
  -> 192.168.1.104:80                   13       80        0     5272        0
  -> 192.168.1.105:80                   14       91        0     5868        0

-stats options are statistics since that article forwarding rules in force

• Conns (connections scheduled) has forwarded the number of connections
• InPkts (incoming packets) into the number of packets
• OutPkts (outgoing packets) the packet number
• InBytes (incoming bytes) into the flow (bytes)
• OutBytes (outgoing bytes) the flow (bytes)

3. ipvsadm -Ln --rate

root@director:~# ipvsadm  -Ln --rate
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port                 CPS    InPPS   OutPPS    InBPS   OutBPS
  -> RemoteAddress:Port
TCP  10.0.0.100:80                       1        4        0      251        0
  -> 192.168.1.104:80                    0        2        0      109        0
  -> 192.168.1.105:80                    0        2        0      142        0

-rate option is to display rate information

• CPS (current connection rate) per second connector
• InPPS (current in packet rate) the number of packets per second
• OutPPS (current out packet rate) the number of packets per second
• InBPS (current in byte rate) into the second flow (bytes)
• OutBPS (current out byte rate) into the second flow (bytes)

---

Although you feel would use ipvsam command, but the production process is rarely used directly ipvsadm direct management of the machine, but through the realization keepalive

Third, the rule of Purity

ipvsadm-save the default rule printed to the screen

root@z2:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.1.102:80 rr
  -> 192.168.1.103:8080           Masq    1      0          3         
  -> 192.168.1.104:8080           Masq    1      0          13        
root@z2:~# ipvsadm-save
-A -t 10.20.1.102:http -s rr
-a -t 10.20.1.102:http -r 192.168.1.103:http-alt -m -w 1
-a -t 10.20.1.102:http -r 192.168.1.104:http-alt -m -w 1

You can use ipvsadm-save redirected to a file, and then use ipvsadm-restore recovery

root@z2:~# ipvsadm  -C
root@z2:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
root@z2:~# ipvsadm-restore  < /data/ipvs.log 
root@z2:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.20.1.102:80 rr
  -> 192.168.1.103:8080           Masq    1      0          0         
  -> 192.168.1.104:8080           Masq    1      0          0