1. Nginx load balancing
1.1 Requirements for building a load balancing service
Share the large-scale concurrent access or data traffic that a single computer cannot bear to multiple node devices and process them separately, reducing the time users wait for a response and improving user experience;
A single heavy-load operation is shared among multiple node devices for parallel processing. After each node device completes the processing, the results are summarized and returned to the user. The system processing capacity is greatly improved.
7*24 hours of service guarantee, any one or more limited back node equipment is down, which cannot affect the business.
1.2 Detailed explanation of Nginx load balancing module
There are two main components for implementing Nginx load balancing: ngx_http_proxy_module (proxy proxy module, which is used to send requests to server nodes or upstream server pools.) ngx_http_upstream_module (load balancing module, which can realize the load balancing function of the website and the health check of nodes .) .
The load balancing function of Nginx relies on the ngx_http_upstream_module module, and the supported proxy methods include proxy_pass, fastcgi_pass, memcached_pass, etc. The ng_http_upstream_module module allows Nginx to define one or more sets of node server groups. When used, the request of the website can be sent to the name of the corresponding Upstream group defined in advance through the proxy_pass proxy method. The specific writing method is "proxy_pass http://www_server_pools " , where www_server_pools is the name of an Upstream node server group.
1.2 Basic configuration of Nginx load balancing
First, configure the Nginx.conf configuration file for two identical web services as follows:
user root;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
keepalive_timeout 65;
server {
listen 80;
server_name bbs.carlton.com;
location / {
root html/bbs;
index index.html index.htm;
}
access_log logs/bbs_access.log main;
}
server {
listen 80;
server_name www.123456.com;
location / {
root html/bbs;
index index.html index.htm;
}
access_log logs/bbs_access.log main;
}
}And also configure the Nginx.conf file of the other server
Use the third server to configure load balancing, and configure the local hosts for domain name resolution
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
upstream server_pools {
server 10.0.137.143:80 weight=1;
server 10.0.137.144:80 weight=1;
}
server {
listen 80;
server_name www.123456.com;
location / {
proxy_pass http://server_pools;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
server {
listen 80;
server_name bbs.carlton.com;
location / {
proxy_pass http://server_pools;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
}
}After the configuration is complete, visit bbs.carlton.com, and the phenomenon of the webpage will keep jumping.
1.3 upstream configuration case
1.3.1 Basic upstream configuration case
upstream www_server_pools{ #upstream is a must have a keyword. The following www_server_pools is the name of an upstream cluster group. You can start it yourself and use this name when calling.
server 10.0.0.7:80 weight=5; #server is a fixed keyword, followed by a domain name or IP. If no port is specified, the default is port 80, and weight represents the weight. The larger the value, the more requests will be allocated. end with a semicolon
server 10.0.0.8:80 weight=10;
}
1.3.2 A more complete upstream configuration case
upstream blog_server_pool {
server 10.0.0.7; #This line of labels is equivalent to the next line.
server 10.0.0.7:80 weight=1 max_fails=1 fail_timeout=10; #This line of label is equivalent to the previous line. The part of this line is the default configuration, and it is okay to not write it.
server 10.0.0.8:80 weight=1 max_fails=2 fail_timeout=20 backup; #server can add many parameters at the end
}
1.3.3 Upstream configuration example using domain name and socket
upstream backend {
server backend2.example.com:8080 #Domain name plus port, forward to the port specified by the backend
server unix:/tmp/backend3; #Specify socket file
}
1.3.4 Description of upstream modules
The content of the upstream module should be protected from the http{} tag configured in nginx.conf, and its default scheduling node algorithm is wrr (weighted round-robin)
1.3.5 upstream module content server tag parameter description
| server tag |
Parameter Description |
| server 10.0.0.8:80 |
The RS configuration behind the load balancing can be an IP or a domain name. If the port is not written, the default is port 80. In high concurrency scenarios, IPs can be replaced with domain names, and load balancing can be done through DNS. |
| weight=1 |
Represents the weight of the server, the default value is 1. The higher the weight number, the higher the proportion of requests accepted. |
| max_fails=1 |
The number of times Nginx failed to connect to the back-end host. This value is used with the three parameters proxy_next_upstream, fastcgi_next_upstream and memcached_next_upstream. When Nginx receives the status code defined by the three parameters returned by the back-end server, it will forward the request to Normal working backend servers, such as 404, 502, 503, and the default value of Max_fails are 1; for enterprise scenarios: 2-3 times are recommended, 1 time for JD.com, and 10 times for Blue Flood, which can be configured according to business requirements. |
| backup |
Hot standby configuration, the hot standby RS will be automatically enabled after all the previously activated RSs fail. This indicates that this server is used as a backup server. If all the primary servers are down, requests will be forwarded to it. Note that when the load scheduling algorithm is ip_hash, the posture of the backend server in the load balancing scheduling cannot be weight and backup. |
| fail_timeout=10s |
After the number of failures defined by max_fails, the interval from the next check is 10s by default; if max_fails is 5, it will check 5 times, if 5 are 502. Then, it will wait 10s according to the value of fail_timeout Go to check again, or only check it once. If 502 persists, it will only be checked every 10s without reloading the nginx configuration. 2-3 seconds for regular business is reasonable, such as 3 seconds for JD.com and 3 seconds for Lanxun, which can be configured according to business needs. |
| down |
This means that the server is never available, this parameter can be used with ip_hash |
1.4 Upstream Module Scheduling Algorithm
1.4.1 Scheduling algorithms are generally divided into two categories
The first type is the static scheduling algorithm, that is, the load balancer allocates according to the rules set by itself, and does not need to consider the situation of the back-end node server. For example, rr, wrr, ip_hash, etc. are all static scheduling algorithms.
The second type is dynamic scheduling algorithm, that is, the load balancer will decide whether to distribute requests according to the current state of the backend nodes. For example, requests with a small number of connections are given priority, and requests with a short response time are given priority. For example: least_conn, fair, etc. are all dynamic scheduling algorithms
1.4.2 Introduce common scheduling algorithms
rr polling (weight query, static scheduling algorithm)
wrr (weight query, static scheduling algorithm)
Adding weights on the basis of the rr polling algorithm is the weighted polling algorithm. When this algorithm is used, the weight is proportional to user access. The larger the weight value, the more requests are forwarded. The weight value can be specified according to the configuration and performance of the server to effectively solve the problem of request allocation caused by the uneven performance of the old and new servers.
ip_hash (static scheduling algorithm)
Each request is allocated according to the hash result of the client IP. When a new request arrives, the client IP is first hashed to a value through the hash algorithm. In subsequent client requests, the hash value of the client IP is as long as If they are the same, they will be assigned to the same server. This scheduling algorithm can solve the session sharing problem of dynamic web pages, but sometimes it will lead to uneven distribution of requests, that is, 1:1 load balancing cannot be guaranteed. In NAT Internet access mode, multiple clients will correspond to one external IP. Therefore, these clients will be assigned to the same node server, resulting in uneven distribution of requests.
fair (dynamic scheduling algorithm)
This algorithm will allocate requests according to the response time of the backend node server, and the shortest response time will be given priority. This is a smarter scheduling algorithm. If you need to use this scheduling algorithm, you must download the Nginx related module upstream_fair
least_conn
The least_conn algorithm will determine the allocation situation according to the number of connections of the back-end nodes, whichever machine has the fewest connections will be distributed
url_hash algorithm
Similar to ip_hash, the request is allocated according to the hash result of the access URL, so that each URL is directed to the same back-end server. The effect is remarkable when the back-end server is a cache server. Add a hash statement to the upstream, and other parameters such as weight cannot be written in the server statement. The hash_method uses the hash algorithm. url_hash distributes requests according to the hash result of the accessed URL, so that each URL is directed to the same backend server, which can further improve the efficiency hit rate of the backend cache server. Nginx itself does not support url_hash. If you need to use this scheduling algorithm, you must install the Nginx hash module package.
Consistent hash algorithm
Consistent hashing algorithms are generally used in scenarios where the proxy backend business serves the cache. The URI or specified string requested by the user is calculated and then dispatched to the backend server. After that, any user looking for the same URI or specified string will It is scheduled to this server, so the content cached by each node in the backend is different. The consistent hash algorithm can solve the problem that when one or several nodes in the backend go down, the cached data will be the least turbulent.
1.5 Explanation of proxy module of reverse proxy
1.5.1 Introduction to proxy_pass directive
The proxy_pass directive belongs to the ngx_http_proxy_module module, which can forward requests to another server. In the actual reverse proxy work, it will match the specified URI through the location function, and then throw the received request matching the matching URI to the proxy_pass. The defined upstream node pool.
Use cases for proxy_pass
Throw the request with name matching the URI to http://127.0.0.1/remote/
location /name/ {
proxy_pass http://127.0.0.1/remote/
}
1.5.2 http proxy module related parameters
| http proxy module related parameters |
illustrate |
| proxy_set_header |
Set the HTTP request header item to be passed to the back-end server node, for example, the server node of the proxy back-end can obtain the real IP address of the user accessing the client. |
| client_body_buffer_size |
Used to specify the client request topic buffer size |
| proxy_connect_timeout |
Indicates the timeout period for the connection between the reverse proxy and the backend node server, that is, the timeout period for initiating a handshake and waiting for a response. |
| proxy_send_timeout |
Indicates the data return time of the proxy backend server, that is, the backend server must transmit all data within the specified time, otherwise, nginx will disconnect the connection. |
| proxy_read_timeout |
Set the time for nginx to obtain information from the proxy's back-end server, indicating that after the connection is successfully established, nginx waits for the response time of the back-end access right, which is actually the time that nginx has entered the back-end queue and waits for processing. |
| proxy_buffer_size |
Set the buffer size. By default, the buffer size is equal to the size set by the proxy_buffers command. |
| proxy_buffers |
Set the number and size of buffers. The response information obtained by nginx from the proxy's back-end server will be placed in the buffer |
| proxy_busy_buffers_size |
Used to set the size of proxy_buffers that can be used when the system is very busy. The official recommended size is proxy_buffers*2 |
| proxy_temp_file_write_size |
Specifies the size of the proxy cache temporary files |
Warm reminder: the official address of the ngx_upsteam_modeule module:
http://nginx.org/en/docs/http/ngx_http_upstream_module.html
2. LVS cluster and Keepalived manage LVS
2.1 Requirements for building a load balancing service
Load Balance (Load Balance) cluster provides a cheap, effective and transparent method to expand the load, bandwidth, increase throughput, strengthen network data processing capacity, and improve network flexibility and availability of network devices and servers.
So in the above case, the enterprise website needs to build a load balancing service?
A single computer cannot withstand large-scale concurrent access or data traffic. At this time, a load balancing cluster needs to be built to distribute the traffic to multiple node devices for separate processing, which reduces the time for users to wait for a response and improves the user experience;
A single heavy-load computing service is shared on multiple node devices for parallel processing. After each node device finishes processing, the results are summarized and returned, and the system processing capacity and efficiency are greatly improved.
2.2 LVS（Linux Virtual Server）介绍
LVS是Linux Virtual Server的简写，意即Linux虚拟服务器，是一个虚拟的服务器集群系统，可以在UNIX/LINUX平台下实现负载均衡集群的功能。该项目在1998年5月由章文嵩博士组织成立，是中国国内最早出现的自由软件项目之一。
LVS技术点小结：
真正实现调度的工具是IPVS，工作在linux内核层面。
LVS自带的IPVS管理工具是ipvsadm。
keepalived实现管理IPVS及负载均衡器的高可用。
Red hat工具Piranha WEB管理实现调度的工具是IPVS
2.3 LVS的ip名词解释
| 名称 |
缩写 |
说明 |
| 虚拟IP地址（Virtual Ip Address） |
VIP |
VIP为Director用于向客户端计算机提供服务的IP地址。比如：www.123456.com域名就要解析到vip上提供服务。 |
| 真实IP地址（Real Server Ip Address） |
RIP |
在集群下面节点上使用的IP地址，物理IP地址。 |
| Director的IP地址（Director Ip Address） |
DIP |
Director用于连接内外网络的IP地址，物理网卡上的IP地址，是负载均衡器上的IP。 |
| 客户端主机IP地址（Client Ip Address） |
CIP |
客户端用户计算机请求集群服务器的IP地址，该地址用做发送给集群的请求的源IP地址 |
2.4 LVS的四种工作模式特点
2.4.1 DR模式的特点
通过调度器LB上修改数据包的目的MAC地址实现转发。注意，源IP地址仍然是CIP，目的IP地址仍然是VIP。
请求的报文经过调度器，而RS响应处理后的报文无需经过调度器LB，因此，并发访问量大时使用效率很高。
因DR模式是通过MAC地址的改写机制实现的转发，因此，所有RS节点和调度器LB只能在一个局域网LAN中（小缺点）
需要注意RS节点的VIP的绑定（ln：vip/32）和ARP抑制问题。
强调下：RS节点的默认网关不需要是调度器LB的DIP，而直接是IDC机房分配的上级路由器的IP（这是RS电邮外网IP地址的情况），理论讲：只要RS可以出网即可，不是必须要配置外网IP。
由于DR模式的调度器仅进行了目的MAC地址的改写，因此，调度器LB无法改变请求的报文的目的端口
当前，调度器LB支持几乎所有的unix，linux系统，但目前不支持windows系统。真实服务器RS节点可以是windows系统。
总的来说DR模式效率很高，但是配置也较麻烦，因此，访问量不是特别大的公司可以用haproxy/nginx取代之。这符合运维的原则：简单、易用、高效。日1000-2000W PV或并发请求1万以下都可以考虑用haproxy/nginx（LVS NAT模式）
直接对外的访问业务，例如：web服务做RS节点，RS最好用公网IP地址。如果不直接对外的服务，例如：Mysql，存储系统RS节点，最好只用内部IP地址。
2.4.2 NAT模式特点
NAT技术将请求的报文（通过DNAT方式改写）和响应的报文（通过SNAT方式改写），通过调度器地址重写然后在转发给内部的服务器，报文返回时再改写成原来的用户请求的地址。
只需要在调度器LB上配置WAN公网IP即可，调度器也要有私有LAN IP和内部RS节点通信。
每台内部RS节点的网关地址（LDIP），这样才能确保数据报文返回时仍然经过调度器LB。
由于请求和比响应的数据报文都经过调度器LB，因此，网站访问量大时调度器LB有较大瓶颈，一般要求最多10-20台节点。
NAT模式支持对IP和端口的转换，即用户请求10.0.0.1:80，可以通过调度器转换到RS节点的10.0。0.2:8080（DR和TUN模式不具备的）
所有NAT内部RS节点只需配置私有LAN IP即可。
由于数据包来回都需要经过调度器，因此，要开启内核转发net，ipv4，ip_forward=1，当然也包括iptables防火墙的forward功能（DR和TUN模式不需要）。
2.4.3 TUN模式的特点
负载均衡器通过把请求的报文通过IP隧道的方式（请求的报文不经过原目的地址的改写包括MAC，而是直接封装成另外的IP报文）转发至真实服务器，而真实服务器将响应处理后直接返回给客户端用户。
由于真实服务器将响应处理后的报文直接返回给客户端用户，因此，最好RS有一个外网IP地址，这样效率才会更高。理论上：只要能出网即可，无需外网IP地址。
由于调度器LB只处理入站请求的报文。因此，此集群系统的吞吐量可以提高10倍以上，但隧道模式也会带来一定的系统开销。TUN模式适合LAN/WAN。
TUN模式的LAN环境转发不如DR模式效率高，而且还要考虑系统对IP隧道的支持问题
所有的RS服务器都要绑定VIP，抑制ARP，配置复杂。
LAN环境一般采用DR模式，WAN环境可以用TUN模式，但是当前在WAN环境下，请求装发更多的被haproxy/nginx/DNS调度等代理取代。因此，TUN模式在国内公司实际应用的已经很少。跨机房应用要么拉光纤成局域网，要么DNS调度，底层数据还得同步。
直接对外的访问业务员，例如：web服务器做RS节点，最好用公网IP地址。不直接对外的业务，例如：Mysql，存储系统RS节点，最好用内部IP地址。
2.4.4 FULLNAT模式的特点
在LVS的FULLNAT转发模式下, LVS对数据包同时做SNAT和DNAT，将数据包的源IP、源端口更换为LVS本地的IP和端口，将数据包的目的IP和目的端口修改为RS的IP和端口，从而不再依赖特定网络拓朴转发数据包。
3. 安装并配置lvs
lb01和lb02操作如下
yum install ipvsadm -y
ln -s /usr/src/kernels/`uname -r` /usr/src/linux3.2 配置负载均衡服务
lb01和lb02操作（10.0.137.143）
ip addr add 10.0.137.142/24 dev eth0 label eth0:0
ipvsadm -C
ipvsadm -A -t 10.0. 137.142:80 -s wrr
ipvsadm -a -t 10.0. 137.142:80 -r 10.0. 137.143:80 -g -w 1
ipvsadm -a -t 10.0. 137.142:80 -r 10.0.0. 137.144 -g -w 1节点web1 和web2 服务器上操作如下
ip addr add 10.0. 137.142/32 dev lo label lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce3.3 使用keepalived管理LVS
lb01和lb02都安装keepalived后，修改lb0的keepalived配置为如下
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0. 137.142/24 dev eth0 label eth0:1
}
}
#ipvsadm -A -t 10.0.0.3:80 -s wrr
virtual_server 10.0. 137.142 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
#ipvsadm -a -t 10.0.0.3:80 -r 10.0.0.7:80 -g -w 1
real_server 10.0. 137.143 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
#ipvsadm -a -t 10.0.0.3:80 -r 10.0.0.8:80 -g -w 1
real_server 10.0. 137.144 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}修改lb02服务器keepalived配置为如下
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL1
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.0. 137.142/24 dev eth1 label eth1:1
}
}
#ipvsadm -A -t 10.0.0.3:80 -s wrr
virtual_server 10.0. 137.142 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
persistence_timeout 50
protocol TCP
#ipvsadm -a -t 10.0.0.3:80 -r 10.0.0.7:80 -g -w 1
real_server 10.0. 137.143 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
#ipvsadm -a -t 10.0.0.3:80 -r 10.0.0.8:80 -g -w 1
real_server 10.0. 137.144 80 {
weight 1
TCP_CHECK {
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
connect_port 80
}
}
}配置完重启keepalived即可