A. LVS Overview
LVS is a workload balancing solution on the four agreements, created by Dr. Zhang Wen-song in 1998. Load balancing is currently widely used model are:
1) work in four protocol (LVS): mainly used for load balancing on Layer 4 protocol, performance compared to the work agreements in seven better. However, the protocol stack is working in the transport layer, for more advanced features of the transport layer lack of support.
2) work in the seven agreements (Nginx, HAproxy): mainly to solve the load balancing (Nginx can also work on the seven agreements in four, you need to add --with-streams parameters at compile time), more suitable for the HTTP service service, mail service and other high-level protocols. Compared to four load-balancing agreement, at the expense of a small amount of performance, providing more targeted services.
Two. LVS operating mode
2.1 Two-stage works:
LVS works and works similar to iptables, they have a framework in the kernel space (Netfilter, ipvs) + management tool user space (iptables, ipvsadm). User-space management tool provides access to our operational rules, we configure rules related to the kernel space frame, to make it work to achieve the corresponding functions.
LVS model involved may be mainly divided into two categories: one is the Director scheduler (for receiving a request from a client, and forwarding device to a backend service provided by real IPVS rules and algorithms), the other is the RealServer (i.e. back-end equipment to provide real services). LVS is mounted on the Director scheduler, located in the kernel space ipvs listening Netfilter the input chain, and where a set of rules, if it is found that the cluster service access, the forced modification request message (here distinguished NAT mode, the DR mode, mode TUN), and immediately sent to the prerouting chain. If the access request is not determined by the cluster service, or network layer protocol in accordance with normal access to the native [determined in accordance with IP + Port].
2.2 Note:
1, because the LVS model relies on the kernel Netfilter framework, and to deal with access to the cluster service request message, forcibly made changes, which can lead to functional failure iptables. LVS and therefore iptables can not be used together.
2,2.4.23 previous version of the kernel does not have built ipvs code, you need to patch. After this version, you can directly use LVS function.
Three. LVS common working model
3.1.NAT model
working principle:
When the monitoring ClientIP access to cluster services, Director scheduler will select a back-end RealServerIP according to the scheduling algorithm, the data packets to convert postrouting chain DNAT (destination address translation). When the rear end RealServer processing request is completed, the response message again via the Client reply Director, for the SNAT (source address conversion) in the chain prerouting Director.
Advantages and disadvantages:
Configurations are easy; drawback is due to go through the data out of the Director, all performance is poor, typically 10 to drive RIP.
NAT model Note:
1) cluster nodes and the Director must be in the same network.
2) RIP address are generally private address, but only for communication with DIP.
3) Director located between Client and RealServer, responsible for all incoming and outgoing communications.
4) To use the DIP cluster nodes as the default gateway.
5) Director supports port mapping.
6) Any operating system can be used as RealServer.
7) Director easily become a bottleneck cluster performance.
3.2.DR model
working principle:
[Director] Director scheduling stage after receiving the request of the Client (client access is Director of VIP), found that the request is access to the cluster service, it will modify the destination MAC address of the request packet and forwards the packet to the chain by postrouting RealServer.
Each processing stage] [RealServer RealServer having the VIP, disposed on the IP alias, encapsulated response packet only, not for communication with the Director. RealServer When receiving the request packet, the data link protocol own MAC address is found, decapsulates the VIP find themselves arranged on an alias, then receives and processes the packet.
[Stage] RealServer RealServer reply received this request packet, the source IP address and destination IP addresses are not modified. Thus when the reply packet, RealServer The IP network layer reply directly Client, does not require the Director.
Advantages and disadvantages:
The advantage is Director scheduler only needs to process the request message, do not handle the response packet, efficiency is greatly improved, can bring hundreds of RealServer; drawback is that NAT configuration relative to model complex.
DR model Note:
1) Each cluster node must be on the same physical network, because with the Director to be forwarded according to the MAC address.
2) RIP can not have a private network address.
3) Director handles only incoming requests, replies response packet to the client directly by the RealServer.
4) RealServer must not be the default gateway pointing Director, to be able to point to access the Internet routing equipment.
5) Director port mapping can not be achieved, only changes the MAC address, TCP / IP are not changed.
6) Most operating systems can be used on RealServer, Server must support hide IP.
Development: ban RealServer way to respond to ARP
1) Fixed MAC address written in the VIP routing device.
2) arptables chain rule definition rule, does not respond arp.
3) kernel parameter: There are two parameters arp_ignore, arp_announce for defining the host response to broadcast arp level and level announcement. VIP requested by the user and incoming network interface is not the same, then arp ignore this message.
arp_announce:
0: any address of the machine any interface communications outwardly; 1: matching the target network as its network communications interface. 2: the matching advertisement to the local network interfaces only. Default 0.
arp_ignore:
0: The machine is equipped to respond to any representation of the address. 1: Only a response indicating the IP address and the interface is configured with the same IP network. Default 0.
3.3.TUN model
working principle:
Upon receiving the Director Client discovery request packet and access to the service is a cluster, the scheduler requests outside the original IP packet, encapsulating an IP packet again and sends it to the external network (the original IP packet again layer is the encapsulated IP packet tunneling), RealServer finally forwarded to the different network domains.
Advantages and disadvantages:
The advantage is the ability to address the needs of off-site disaster recovery; drawback is the Director and RealServer device supports tunneling, configure complex;
TUN model Note:
1) RealServer across the Internet.
2) RealServer The RIP must be public addresses.
3) Director only process incoming requests, there RealServer response packets sent to the client.
4) only supports the tunneling protocol of the OS can be used for RealServer.
5) does not support port mapping.
Four. LVS configuration details (Centos7)
4.1.NAT model configuration
network layout
NAT mode
VIP:
192.168.100.150 Outer
DIP:172.16.100.10 inner
rip1：
172.16.100.11 inner
rip2：
172.16.100.12 inner
Configuring RealServer
1 # install and configure nginx 2 yum the install nginx -Y
Director configuration
1 # 安装ipvsadm工具 2 yum install ipvsadm -y 3 # 关闭ICMP重定向 4 echo 1 > /proc/sys/net/ipv4/ip_forward 5 echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects 6 echo 0 > /proc/sys/net/ipv4/conf/defaults/send_redirects 7 echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects 8 echo 0 > /proc/sys/net/ipv4/conf/eth1/send_redirects 9 # director设置ipvsadm 10 ipvsadm -A -t 192.168.100.150:80 -s rr 11 ipvsadm -a -t 192.168.100.150:80 -r 172.16.100.11:80 -m 12 ipvsadm -a -t 192.168.100.150:80 -r 172.16.100.12:80 -m
4.2.DR mode configuration
network layout
Director:
VIP: 192168100150
dip: 192168100155
rip1:
lo: 0 192 168 100 150 (VIP)
rip 192168100151
rip2
him: 0 192 168 100 150 (VIP)
rip 192168100152
Configuring RealServer
# Set the kernel parameters, disable respond to the ARP packet echo " . 1 " > / proc / SYS / NET / IPv4 / the conf / LO / arp_ignore echo " 2 " > / proc / SYS / NET / IPv4 / the conf / LO / arp_announce echo " . 1 " > / proc / SYS / NET / IPv4 / the conf / All / arp_ignore echo " 2 " > / proc / SYS / NET / IPv4 / the conf / All / arp_announce # configure alias the VIP the ifconfig LO: 0 192.168 . 100.150 Broadcast 192.168 . 100.150 Netmask 255.255 . 255.255 up # Add a default route to the gateway before the Director, due to the experimental environment using NAT mode, it has a good job points to a default gateway, where it is no longer configured
Director configuration
# Install Nginx and configure yum install nginx - the y- # configure port forwarding echo 1 > / proc / SYS / NET / ipv4 / ip_forward # Configure VIP ifconfig ens33: 0 192.168 . 100.150 Broadcast 192.168 . 100.150 Netmask 255.255 . 255.255 up # added to RealServer routing route the Add -host 192.168 . 100.150 dev ens33: 0 # configure IPVS the ipvsadm -A -t 192.168 . 100.155 : 80 - S RR the ipvsadm-a -t 192.168.100.155:80 -r 192.168.100.151:80 -g ipvsadm -a -t 192.168.100.155:80 -r 192.168.100.152:80 -g