I. Introduction
Linux cluster from the functions can be divided into two categories: high-availability clustering and load balancing clusters. HA has been explained on a high availability cluster, this section explains load balancing cluster.
Cluster load balancing (Load Balance Cluseter, referred to as LB), it is to allow multiple servers to a balanced bearing pressure, is a technology cluster server or network device. The specific load balancing service (network services, network traffic, etc.) to share to multiple servers or network devices, which can solve the problem of network congestion, resulting in improved business processing capabilities, to provide users with access to more consistent quality. Load balancing cluster of open source software LVS, Keepalived, Haproxy, Nginx and so on, of course, the commercial load balancing hardware devices, such as F5, NetScaler and so on. Commercial load balancing solution is relatively expensive, mainly on open source software LVS here.
Two, LVS Profile
LVS (Linux Virtual Server, Linux Virtual Server), by our technical experts Zhang Wen-song developed by well-known open-source software, which is a software-based 4-layer (OSI network 7 layer model), Nginx based on Layer 7, Haproxy can either use for 4 layer, layer 7 may also be used. keepalived load balancing feature is the LVS. LVS simple working principle is the VIP user request LVS, LVS algorithm according to the forwarding mode and the forwarding the request to the back-end servers, back-end server receives a request, returned to the user process. For users, see the Web application specific backend.
LVS (4 layer) load balancing is a TCP / IP routing and forwarding do, Web services are 80 ports, it can be distributed to other communication port. For example, MySQL LVS load balancing can also be used to achieve, and Nginx not support this feature, only supports HTTP, HTTPS, Mail; and Haproxy also supports load balancing of TCP such as MySQL.
In comparison, the LVS this 4 layer structure is more stable, can carry higher amounts complicated, and Nginx layer structure 7 that is more flexible and can achieve more individual needs.
Simple to understand what software load balancing:
1) The so-called hierarchical load balancing, are based on the model of the network is. Four is the IP-based load balancing and port, seven load balancing is based on the application of information such as the URL. So simply put four load balancing is through IP and port to receive requests and then distributed to the real server, seven are receiving requests by URL or host name, and then distributed to the real server.
2) achieved seven is also the basis of the four is achieved, there can be no four-seven. On the seventh floor can do many things, such as a mobile phone or PC may be browser based on seven categories of distinction, WEB server will be divided into two groups, the special mobile phone terminal site.
3) for the client, the client seems to be the same host access. In fact, in order to have a better user experience, starting from the intelligent DNS, based on the client IP source from the domain name resolves to a client or a server closest access to the most rapid one server, but the contents of these clients are not feel the client felt only visit the site soon.
LVS program consists of two parts, including ipvs and ipvsadm.
1) ipvs (ip virtual server) : a piece of code work in kernel space, called ipvs, the code went into effect achieved scheduling.
2) ipvsadm: another period of work in user space, called ipvsadm, is responsible for the preparation ipvs core framework of rules that define who is a cluster service, and who is the real back-end server (Real Server).
LVS related terms
1) DS: Director Server. Load balancing refers to the front end node.
2) RS: Real Server. The back-end server real work.
3) VIP: user request directly to the outside, as the target IP address of the user request.
4) DIP: IP address of the Director Server IP, and is mainly used for internal communication with the host.
5) RIP: Real Server IP, IP address back-end server.
6) CIP: Client IP, IP address access client.
2.1 LVS mode
LVS There are three common modes, namely NAT mode, IP Tunnel mode and DR mode.
LVS architecture has a central role called a distributor (Load Balance, also known as Director, abbreviated as Dir), used to distribute the user's request, there are many real servers to handle user requests (Real Server, referred to as RS).
LVS NAT mode by means of the iptables nat table to achieve the user's requests arrive distributor, through a preset iptables rules, forwards the request packet to the rear end of the RS. RS is set within the network gateway dispenser IP, User request packet through the entire dispenser, the dispenser of the mode becomes the bottleneck. Scheduler has two IP, is a public network IP, within a network IP, the only real servers within a network IP, this model can save public network IP.
LVS IP Tunnel mode requires a public IP network disposed on the distributor and RS all, we call it the VIP, the distributor receives the request packet will be processing the data packet, the destination IP IP to the RS, Thus data packets on to the RS. After the RS receives the data packet, restores the original data packet, such as the destination IP on the VIP, the VIP is the own local, so the RS will process the request and returns the response directly to the user results.
LVS DR mode also requires a public IP configuration on the dispenser and all the RS, which is VIP. And different IP Tunnel, it will MAC address modified data packet is the MAC address of the RS. After the RS receives the data packet, restores the original data packet, such as the destination IP on the VIP, the VIP is the own local, so the RS will process the request and returns the response directly to the user results.
2.2 LVS Scheduling Algorithm
The dispatcher sent by the client requests evenly distributed to the back-end is really the server, which rely on predefined amount scheduling algorithm to achieve, LVS in the kernel main achievement at ten kinds of scheduling algorithms.
1. round robin scheduling
Round robin scheduling (Round Robin, referred to as RR) algorithm is to sequentially and cyclically in the manner of a request to schedule a different server, the algorithm is the biggest feature is simple. Polling algorithm assumes that all the capabilities of the server processes the request are the same, the scheduler will average all requests assigned to each real server.
2. Weighted round robin scheduling
Weighted round robin (Weight Round Robin, referred WRR) algorithm is an optimization primarily supplemental polling algorithm, the LVS will consider the performance of each server, and to add a weight to each server, to the server if the value of A 1, to the server B is 2, then the scheduler schedules the request to the server B, server a would be twice. The higher the weight of the server, the more processing the request.
3. The minimum connection scheduling
The minimum connection scheduling (Least Connections, referred to as LC) algorithm is to assign a new connection request to the server with the smallest number of currently connected. The minimum connection scheduling is a dynamic scheduling algorithm that estimates the number of the server by connecting to the server is currently active. Scheduler needs to record the number of connections to each server has been established, when a request is dispatched to a server, which is 1 plus the number of connections; when the connection is lost or a timeout, which is 1 minus the number of connections.
(Real server cluster systems have similar performance, with a minimum connection scheduling algorithm can better balance the load.)
4. weighted least connections scheduling
Weighted least connections (Weight Least Connections, referred WLC) scheduling algorithm is the smallest connector superset corresponding weight represents the processing performance of each server. The default value of 1 to the server, the system administrator can dynamically set the weight of the server. Weighted least-connection scheduling in scheduling a new connection so that the server has been established as a value proportional to the number of connections and their weights. Queries scheduler may automatically load the real server, and dynamically adjusts the weight.
The local connection based minimum
Based on local minimum connection scheduling (Locality-Based Least Connections, referred LBLC) load balancing scheduling algorithm is a target IP address request packet, the key for Cache cluster system, because in Cache Clusters customers request packet destination IP address change. It is assumed that any backend server can handle a request for any, the algorithm is designed in a substantially balanced load the server will request the same destination IP address are dispatched to the same server, each server to improve locality and Cache hit rate, so as to enhance the processing capability of the whole cluster system. LBLC scheduling algorithm first find the IP address of the target server according to the most recently used destination IP address request, if the server is available and is not overloaded, send a request to the server; if the server does not exist, or if the server has a server is overloaded and workload in half, using the principle of "least connections' to select an available server, sends a request to the server.
6. band replication connection based minimum Locality
With a copy of the Locality-Based Least-Connection (Locality-Based Least Connections with Replication, referred LBLCR) algorithm is load balancing target IP address, the key for Cache cluster system, which differs from the LBLC algorithm is that it wants to maintain from a target IP address mapping to a group of servers, and LBLC algorithm maintains a mapping from a target IP address to a server. Press 'minimum connection' principle selected from eleven servers in the server group, if the server is not overloaded, send a request to the server; if the server is overloaded, press 'minimum connection' a principle selected from the entire server cluster the server is then added to the server group, sends a request to the server. Meanwhile, when the server group for some time not been modified, the busiest server is removed from the server group, in order to reduce the degree of replication.
7. destination address hashing scheduling
Scheduling target address hash (Destination Hashing, abbreviated DH) algorithm according to the request destination IP address as the hash key (Hash Key) to find the corresponding server from the list of hash static allocation, if the server is not available and overloaded, the request is sent to the server, otherwise empty.
8. The source address hashing scheduling
Scheduling the source address hash (Source Hashing, referred to SH) algorithm according to the source IP address of the request, as the hash key (Hash Key) to find the corresponding server from the list of hash static allocation, if the server is not available and overloaded, the request is sent to the server, otherwise empty. Hash function with the hash of the destination address it uses the same scheduling algorithm, which is substantially similar to the target address hash algorithm process scheduling algorithm.
9. shortest desired delay
The shortest expected delay scheduling (Shortest Expected Delay, referred to as SED) algorithm Algorithm WLC. To give an example, the right to ABC three servers weight 1,2,3, respectively. So if you use WLC algorithm, then a new request comes in it may give any one of ABC. We will perform a computation algorithm after using SED
A: (1 + 1) / 1 = 2 B: (1 + 2) / 2 = 3/2 C: (1 + 3) / 3 = 4/3 puts the request to the server which yields a minimum calculation result.
10. The queue scheduling Minimum
Minimum queue scheduling (Never Queue, referred NQ) algorithm, without queue. If the number of connections is equal to 0 realserver directly assigned past, no operation is performed SED.
Above algorithm, which is more commonly used first four, but also easier to understand.
Three, LVS NAT mode to build
3.1 Environment Preparation
Preparation 3 servers, one of which as the distributor (also called scheduler, referred to the Dir), the other two are Real Server, as the user requests the server process.
Hostname masternode, distributor requires two NICs, IP 192.168.0.110 card 1 is within the network (VMWare bridged network), IP 192.168.56.110 card 2 is outside the network (VMWare host-only host mode only).
Hostname datanode1, Real Server 1 IP is 192.168.0.111, the gateway to the network masternode set the IP address 192.168.0.110.
Hostname datanode2, Real Server 1 IP is 192.168.0.112, the gateway to the network masternode set the IP address 192.168.0.110.
Three servers were set up for the IP address and host name, and turn off the firewall.
To three servers are installed iptables tool, otherwise it will appear the following prompt (three server are executed only masternode command listed here, the same as the other two rs command):
[root@masternode ~]# service iptables save The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, force-reload, status). For other actions, please try to use systemctl. [root@masternode ~]# yum install -y iptables-services
To three servers are installed, start the iptables services after installation and add the boot from the start, as follows (all three server to perform, listed here only masternode command, the same as the other two rs command):
[root@masternode ~]# systemctl start iptables.service
[root@masternode ~]# systemctl enable iptables.service
Empty iptables rules, and save the blank rule, and then be sure to close Selinux, as follows (all three server to perform, listed here only masternode command, the same as the other two rs command):
[root@masternode ~]# iptables -F [root@masternode ~]# iptables -t nat -F [root@masternode ~]# services iptables save [root@masternode ~]# getenforce Disabled
Make sure that both RS intranet server gateway to the distributor within the network.
3.2 Setting dispenser
Dir dispenser mounted above ipvsadm, ipvsadm is the core tool for achieving LVS function, as follows:
[root@masternode ~]# yum install -y ipvsadm
LVS are conducted in the form of a script, and easy maintenance. Dir write a script in the server above, as follows:
[root @ masternode sbin] # vim / usr / local / sbin / lvs_nat. SH # ! / bin / bash on # director distribution server open routing forwarding echo 1 > / proc / SYS / NET / ipv4 / ip_forward # close ICMP redirect echo 0 > / proc / SYS / NET / ipv4 / conf / All / send_redirects echo 0 > / proc / SYS / NET / ipv4 / conf / default / send_redirects # pay attention to distinguish the name of the card, here are two network cards enp0s8 and enp0s9 echo 0 > / proc / SYS / NET / IPv4 / the conf / enp0s8 / send_redirects echo 0 > / proc / SYS / NET / IPv4 / the conf / enp0s9 / send_redirects # Director distributor disposed NAT firewall iptables -t NAT - F. iptables-t nat -X iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE # director 分发器设置ipvsadm IPVSADM='/usr/sbin/ipvsadm' $IPVSADM -C $IPVSADM -A -t 192.168.56.110:80 -s wlc $IPVSADM -a -t 192.168.56.110:80 -r 192.168.0.111:80 -m -w 1 $IPVSADM -a -t 192.168.56.110:80 -r 192.168.9.112:80 -m -w 1 [root@masternode sbin]# chmod 755 /usr/local/sbin/lvs_nat.sh
Do not start Nginx in Dir above, if there is to start Nginx service, you need to turn it off: killall nginx
Setting Real Server 3.3
Two RS Nginx mounted above, may be used to install source package, you may be installed with yum, but, when used yum installed, since there is no source centos official Nginx package, install epel extended source in advance, the installation process may refer to a section.
Two RS set different home page content, easy to distinguish, the default Web site after yum install root directory is / usr / share / nginx / html, as follows:
[root@datanode1 html]# echo 'Real Server 1' > /usr/share/nginx/html/index.html [root@datanode2 html]# echo 'Real Server 2' > /usr/share/nginx/html/index.html
Start both the RS respectively Nginx, as follows:
[root@datanode1 html]# systemctl start nginx.service [root@datanode1 html]# ps aux |grep nginx root 13199 0.0 0.0 120796 2092 ? Ss 18:27 0:00 nginx: master process /usr/sbin/nginx nginx 13200 0.0 0.0 123264 3552 ? S 18:27 0:00 nginx: worker process nginx 13201 0.0 0.0 123264 3552 ? S 18:27 0:00 nginx: worker process root 13203 0.0 0.0 112704 956 pts/0 S+ 18:27 0:00 grep --color=auto nginx [root@datanode2 html]# systemctl start nginx.service [root@datanode2 html]# ps aux |grep nginx root 12408 0.0 0.0 120796 2096 ? Ss 18:27 0:00 nginx: master process /usr/sbin/nginx nginx 12409 0.0 0.0 123264 3556 ? S 18:27 0:00 nginx: worker process nginx 12410 0.0 0.0 123264 3556 ? S 18:27 0:00 nginx: worker process root 12412 0.0 0.0 112704 960 pts/0 S+ 18:28 0:00 grep --color=auto nginx
3.4 Running View
Dir run scripts at the top, as follows:
[root@masternode ~]# sh /usr/local/sbin/lvs_nat.sh [root@masternode ~]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 192.168.56.110:80 wlc -> 192.168.0.111:80 Masq 1 0 0 -> 192.168.0.112:80 Masq 1 0 0 [root@masternode ~]# ipvsadm -Lnc IPVS connection entries pro expire state source virtual destination
Wherein ipvsadm -Ln can view the forwarding rule LVS-NAT mode, ipvsadm -Lnc IPVS can view scheduling state.
3.5 Test LVS NAT mode
Enter the VIP address in the address bar of your browser to access 192.168.56.110 load balancing system.
Four, LVS DR mode is set up
DR mode is still prepared three servers, each server requires only one network adapter, an IP, the 192.168.0 network to the public network IP, VIP set to 192.168.0.190. Configuration changes before two Real Server gateway to the original 192.168.0.1.
Set Dir 4.1
LVS is set DR mode, create a script in Dir above, and set the rules, as follows:
[the root @ masternode sbin] # Vim / usr / local / sbin / lvs_dr. SH # ! / bin / the bash echo . 1 > / proc / SYS / NET / IPv4 / ip_forward IPV = / usr / sbin / the ipvsadm VIP = 192.168 . 0.190 RS1 = 192.168 . 0.111 rs2 = 192.168 . 0.112 # set the virtual NIC, called enp0s9: 2 , IP is 192. The 168.0 . 190 ifconfig enp0s9: 2 $ vip vip Netmask Broadcast $ 255.255 . 255.255 up # set the gateway route add-host $vip dev enp0s9:2 $ipv -C $ipv -A -t $vip:80 -s wlc $ipv -a -t $vip:80 -r $rs1:80 -g -w 1 $ipv -a -t $vip:80 -r $rs2:80 -g -w 1
Lvs_dir.sh run the script, the script takes effect.
[root@masternode sbin]# sh /usr/local/sbin/lvs_dr.sh
4.2 Setting RS
Real Server on both servers also need to write a script, the code is as follows:
[root @ datanode1 sbin] # vim / usr / local / sbin / lvs_rs. SH # ! / bin / bash vip = 192.168 . 0.190 # The vip bound to lo, in order to make rs directly the results back to the client ifconfig LO: 0 $ vip vip Netmask Broadcast $ 255.255 . 255.255 up route the Add -host $ vip LO: 0 # following kernel parameters to change the ARP, the purpose is to allow the smooth rs mac address sent to the client echo " 1 " > / proc / SYS / NET / IPv4 / the conf / LO / arp_ignore echo " 2 " > / proc / SYS / NET / IPv4 / the conf / LO / arp_announce echo " . 1 " > /proc/sys/net/ipv4/conf/all/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce
Run the script, as follows:
[root@datanode1 sbin]# sh /usr/local/sbin/lvs_rs.sh [root@datanode2 sbin]# sh /usr/local/sbin/lvs_rs.sh
Use route -n command to view the routing table.
4.3 Testing of the DR mode LVS
Enter the VIP address in the address bar of your browser 192.168.0.190 to access the load-balancing system.