Theory + Experiment-How to use Haproxy to build a Web cluster

1. Haproxy overview

Haproxy is currently a popular cluster scheduling tool

1.1 Comparison of Haproxy with LVS and Nginx

• LVS has the best performance, but is relatively complicated to build
• The upstream module of Nginx supports the cluster function, but the health check function of the cluster nodes is not strong, and the performance is not as good as Haproxy.

1.2 Common Web Cluster Scheduler

• The current common Web cluster scheduler is divided into software and hardware
• Software usually uses open source LVS, Haproxy, Nginx
• The most commonly used hardware is F5, and many people use some domestic products, such as Barracuda, NSFOCUS, etc.

1.3 Haproxy application analysis

Although LVS has strong anti-load ability in enterprise applications, it has shortcomings

• LVS does not support regular processing and cannot achieve dynamic and static separation
• For large websites, the implementation and configuration of LVS are complicated and the maintenance cost is relatively high

Haproxy is a software that provides high availability, load balancing, and proxy based on TCP and HTTP applications

• Especially suitable for Web sites with heavy load
• Running on current hardware can support tens of thousands of concurrent connection requests

Two, Haproxy scheduling algorithm

Haproxy supports a variety of scheduling algorithms, the most commonly used are three: RR (Round Robin), LC (Least Connections), SH (Source Hashing)

2.1 RR（Round Riobin）

• The RR algorithm is the simplest and most commonly used algorithm, namely round-robin scheduling
• Understanding example
  ◆ There are three nodes A, B, C
  ◆ The first user access will be assigned to node A
  ◆ The second user access will be assigned to node B
  ◆ The third user access will be assigned to node C
  ◆ Fourth A user access continues to be assigned to node A, polling to allocate access requests to achieve load balancing

2.2 LC（Least Connections）

• The LC algorithm is the minimum connection algorithm, which dynamically allocates front-end requests according to the number of back-end node connections
• Understanding example
  ◆ There are three nodes A, B, C, and the number of connections of each node are A:4, B:5, C:6.
  ◆ The first user connection request will be assigned to A, and the number of connections will become A : 5, B: 5, C: 6
  ◆ The second user request will continue to be allocated to A, and the number of connections becomes A6, B: 5, C: 6; a new request will be allocated to B each time The new request is assigned to the client with the smallest number of connections
  ◆ Because the number of connections of A, B, and C will be dynamically released in actual situations, it is difficult to have the same number of connections
  ◆ This algorithm is greatly improved compared to the rr algorithm. An algorithm that is currently used more

2.3 SH（Source Hashing）

• SH is based on the source access scheduling algorithm. This algorithm is used in some scenarios where Session sessions are recorded on the server side. Cluster scheduling can be done based on the source IP, Cookie, etc.
• Understanding example
  ◆ There are three nodes A, B, and C. The first user is assigned to A for the first visit, and the second user is assigned to B for the first visit.
  ◆ When the first user visits for the second time, it will continue. Assigned to A, the second user will still be assigned to B for the second visit. As long as the load balancing scheduler does not restart, the first user's access will be assigned to A, and the second user's access will be assigned to B. Realize cluster scheduling
  ◆ The advantage of this scheduling algorithm is to achieve session retention, but when some IP access is very large, it will cause load imbalance, and some nodes have excessive access, which affects business use

Three, experiment

3.1 Experiment preparation

Host	operating system	IP address	Main software
Haproxy server	CentoS7.6	192.168.100.21	haproxy-1.5.19.tar.gz
Nginx server 1	CentoS7.6	192.168.100.22	Nginx-1.12.2.tar.gz
Nginx server 2	CentoS7.6	192.168.100.23	Nginx-1.12.2.tar.gz
Storage server	CentoS7.6	192.168.100.24	nfs-utils rpcbind

3.2 Step

3.2.1 Debug storage server (192.168.100.24)

'rpm -q nfs-utils          ###如果没装，yum -y install nfs-utils
 rpm -q rpcbind            ###如果没装，yum -y install rpcbind'
[root@localhost ~]# systemctl start nfs
[root@localhost ~]# systemctl start rpcbind
[root@localhost ~]# vi /etc/exports
/opt/51xit 192.168.100.0/24 (rw,sync)
/opt/52xit 192.168.100.0/24 (rw,sync)
[root@localhost ~]# systemctl restart nfs
[root@localhost ~]# systemctl restart rpcbind
[root@localhost ~]# systemctl enable nfs
[root@localhost ~]# systemctl enable rpcbind
[root@localhost ~]# mkdir /opt/51xit /opt/52xit
[root@localhost ~]# echo "this is www.51xit.com" >/opt/51xit/index.html
[root@localhost ~]# echo "this is www.52xit.com" >/opt/52xit/index.html

3.2.2 Compile and install Nginx server 1 (192.168.100.22)

[root@localhost ~]#yum -y install pcre-devel zlib-devel gcc-c++
[root@localhost ~]# useradd -M -s /sbin/nologin nginx
[root@localhost ~]# cd /opt
[root@localhost opt]# ll
total 960
-rw-r--r-- 1 root root 981687 Sep 24 15:58 nginx-1.12.2.tar.gz
[root@localhost ~]# tar zxvf nginx-1.12.2.tar.gz
[root@localhost ~]# cd nginx-1.12.2
[root@localhost nginx-1.12.2]# 
./configure \
--prefix=/usr/local/nginx \
--user=nginx \
--group=nginx
[root@localhost nginx-1.12.2]# make
[root@localhost nginx-1.12.2]# make install
[root@localhost nginx-1.12.2]# ln -s /usr/local/nginx/sbin/nginx /usr/local/sbin/
[root@localhost nginx-1.12.2]# ls -l /usr/local/sbin/nginx
lrwxrwxrwx 1 root root 27 5 月 16 16:50 /usr/local/sbin/nginx -> /usr/local/nginx/sbin/nginx
[root@localhost nginx-1.12.2]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

'■启动、 停止 Nginx
killall -1 nginx                                                     ####安全重启
killall -3 nginx                                                     ###停止服务
如果出现: -bash: killall: command not found'
[root@localhost nginx-1.12.2]# yum -y install psmisc
[root@localhost nginx-1.12.2]# nginx                                ####启动
[root@localhost nginx-1.12.2]# yum -y install net-tools
[root@localhost nginx-1.12.2]# netstat -anpt | grep nginx
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      20107/nginx: master
'###添加 Nginx 系统服务###'
[root@localhost nginx-1.12.2]# vim /lib/systemd/system/nginx.service
[Unit]
Description=nginx                                                 ####描述
After=network.target                                              ####描述服务类别
[Service]
Type=forking                                                      ####后台运行形式
PIDFile=/usr/local/nginx/logs/nginx.pid                           ####PID 文件位置
ExecStart=/usr/local/nginx/sbin/nginx                             ####启动服务
ExecReload=/usr/bin/kill -s HUP $MAINPID                         ####根据 PID 重载配置
ExecStop=/usr/bin/kill -s QUIT $MAINPID                          ####根据 PID 终止进程
PrivateTmp=true
[Install]
WantedBy=multi-user.target
[root@localhost nginx-1.12.2]# chmod 754 /lib/systemd/system/nginx.service
[root@localhost nginx-1.12.2]# systemctl status nginx
● nginx.service - nginx
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
[root@localhost nginx-1.12.2]# systemctl start nginx.service
Job for nginx.service failed because the control process exited with error code. See "systemctl status nginx.service" and "journalctl -xe" for details.
[root@localhost nginx-1.12.2]# killall -3 nginx
[root@localhost nginx-1.12.2]# systemctl start nginx.service
[root@localhost nginx-1.12.2]# systemctl status nginx.service
● nginx.service - nginx
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-09-24 17:08:01 CST; 6s ago
  Process: 77233 ExecStart=/usr/local/nginx/sbin/nginx (code=exited, status=0/SUCCESS)
 Main PID: 77234 (nginx)
    Tasks: 2
   CGroup: /system.slice/nginx.service
           ├─77234 nginx: master process /usr/local/nginx/sbin/nginx
           └─77235 nginx: worker process

Sep 24 17:08:01 localhost.localdomain systemd[1]: Starting nginx...
Sep 24 17:08:01 localhost.localdomain systemd[1]: PID file /usr/local/nginx/l...
Sep 24 17:08:01 localhost.localdomain systemd[1]: Started nginx.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost nginx-1.12.2]# systemctl enable nginx.service
Created symlink from /etc/systemd/system/multi-user.target.wants/nginx.service to /usr/lib/systemd/system/nginx.service
'####刷下命令，服务正常工作了###
killall -3 nginx                                          ###停止服务
systemctl start nginx.service
###上面命令一刷，下面就可以正常使用了###
systemctl start nginx.service
systemctl stop nginx.service
systemctl reload nginx.service
systemctl restart nginx.service
systemctl status nginx.service'
###安装httpd 挂载测试页###
[root@localhost nginx-1.12.2]# yum -y install nfs-utils
[root@localhost nginx-1.12.2]# systemctl start nfs
[root@localhost nginx-1.12.2]# systemctl enable nfs
Created symlink from /etc/systemd/system/multi-user.target.wants/nfs-server.service to /usr/lib/systemd/system/nfs-server.service.
[root@localhost nginx-1.12.2]# showmount -e 192.168.100.24
Export list for 192.168.100.24:
/opt/52xit 192.168.100.0/24
/opt/51xit 192.168.100.0/24
[root@localhost nginx-1.12.2]# mount 192.168.100.24:/opt/51xit /usr/local/nginx/html/                         ###临时挂载
[root@localhost nginx-1.12.2]# vi /etc/fstab                ###永久挂载，需要重启
192.168.100.24:/opt/51xit/ /usr/local/nginx/html/ nfs defaults,_netdev 0 0
[root@localhost nginx-1.12.2]# init 6

Web page login: 192.168.100.22 verification

3.2.3 Compile and install Nginx server 2 (192.168.100.23)

The operation is the same as 3.2.2 , just change 51xit to 52xit , you can refer to 3.2.2

Web page login: 192.168.100.23 verification

3.2.4 Configure Haproxy server (192.168.100.21)

[root@localhost ~]# yum -y install pcre-devel bzip2-devel gcc gcc-c++
[root@localhost ~]# cd /opt
[root@localhost opt]# tar xzvf haproxy-1.4.24.tar.gz 
[root@localhost opt]# cd haproxy-1.4.24/
[root@localhost haproxy-1.4.24]# make TARGET=linux26
[root@localhost haproxy-1.4.24]# make install
[root@localhost haproxy-1.4.24]# mkdir /etc/haproxy
[root@localhost haproxy-1.4.24]# cp examples/haproxy.cfg /etc/haproxy/
[root@localhost haproxy-1.4.24]# vi /etc/haproxy/haproxy.cfg
global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 notice
        #log loghost    local0 info
        maxconn 4096
        #chroot /usr/share/haproxy
        uid 99
        gid 99
        daemon
        #debug
        #quiet

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        #redispatch
        maxconn 2000
        contimeout      5000
        clitimeout      50000
        srvtimeout      50000

listen  webcluster 0.0.0.0:80
        option httpchk GET /index.html
        balance roundrobin
        server inst1 192.168.100.22:80 check inter 2000 fall 3
        server inst2 192.168.100.23:80 check inter 2000 fall 3
'###上述配置文件解释###
1、Haproxy配置文件通常分为三个部分
global：为全局配置
defaults：为默认配置
listen：为应用组件配置
global配置参数
log 127.0.0.1 local0：配置日志记录，配置日志记录,local0为日志设备,默认存放到系统日志
log 127.0.0.1 local1 notice：notice为日志级别,通常有24个级别
maxconn 4096：最大连接数
uid 99：用户uid          gid 99：用户gid

2、defaults配置项配置默认参数，一般会被应用组件继承，如果在应用组件中没有特别声明，将安装默认配置参数设置
log global：定义日志为global配置中的日志定义
mode http：模式为http
option httplog：采用http日志格式记录日志
option dontlognull ：保证HAProxy不记录上级负载均衡发送过来的用于检测状态没有数据的心跳包
retries 3：检查节点服务器失败连续达到三次则认为节点不可用
maxconn 2000：最大连接数
contimeout 5000：连接超时时间
clitimeout 50000：客户端超时时间
srvtimeout 50000：服务器超时时间

3、listen配置项目一般为配置应用模块参数
listen  appli4-backup 0.0.0.0:10004：定义一个appli4-backup的应用
option  httpchk /index.html：检查服务器的index.html文件
option  persist ：强制将请求发送到已经down掉的服务器
balance roundrobin：负载均衡调度算法使用轮询算法
server  inst1 192.168.114.56:80 check inter 2000 fall 3：定义在线节点
server  inst2 192.168.114.56:81 check inter 2000 fall 3 backup：定义备份节点'
[root@localhost haproxy-1.4.24]# cp examples/haproxy.init /etc/init.d/haproxy
[root@localhost haproxy-1.4.24]# chmod 755 /etc/init.d/haproxy
[root@localhost haproxy-1.4.24]# chkconfig --add haproxy
[root@localhost haproxy-1.4.24]# ln -s /usr/local/sbin/haproxy /usr/sbin/haproxy
[root@localhost haproxy-1.4.24]# service haproxy start
[root@localhost haproxy-1.4.24]# systemctl stop haproxy.service 
[root@localhost haproxy-1.4.24]# systemctl start haproxy.service

Web page login: 192.168.100.21 verification
switch, you will find different website pages, indicating that load balancing has been achieved

Four, Haproxy log file

[root@localhost haproxy-1.4.24]# mkdir -p var/log/haproxy                 ###创建日志文件存放点
[root@localhost haproxy-1.4.24]# vi /etc/haproxy/haproxy.cfg
global                        ###下面log开头的 统统去掉换下面的配置
        log /dev/log    local0 info
        log /dev/log    local1 notice
[root@localhost haproxy-1.4.24]# systemctl restart haproxy.service
[root@localhost haproxy-1.4.24]# touch /etc/rsyslog.d/haproxy.conf
[root@localhost haproxy-1.4.24]# vi /etc/rsyslog.d/haproxy.conf 
if ($programname == 'haproxy' and $syslogseverity-text == 'info') then -/var/log/haproxy/haproxy-info.log
& ~
if ($programname == 'haproxy' and $syslogseverity-text == 'notice') then -/var/log/haproxy/haproxy-notice.log
& ~
[root@localhost haproxy-1.4.24]#  systemctl restart rsyslog.service 
[root@localhost haproxy]# tail -f /var/log/haproxy/haproxy-info.log
Sep 24 17:23:14 localhost haproxy[65722]: 192.168.100.2:62258 [24/Sep/2020:17:22:21.206] webcluster webcluster/inst1 0/0/0/1/52926 200 1700 - - cD-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Sep 24 17:23:54 localhost haproxy[65722]: 192.168.100.2:62275 [24/Sep/2020:17:23:41.993] webcluster webcluster/inst2 0/0/0/0/12236 200 258 - - ---- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
Sep 24 17:24:53 localhost haproxy[65722]: 192.168.100.2:62293 [24/Sep/2020:17:24:03.530] webcluster webcluster/inst1 0/0/0/1/50041 200 984 - - cD-- 0/0/0/0/0 0/0 "GET / HTTP/1.1"
[root@localhost haproxy-1.4.24]# cd /var/log/haproxy
[root@localhost haproxy]# ll
total 4
-rw------- 1 root root 538 Sep 24 17:24 haproxy-info.log

Web page login: 192.168.32.21
and then check the continuous generation of log files

Five, Haproxy parameter optimization

As the load on the corporate website increases, haproxy parameter optimization is very important:

• maxconn: Maximum number of connections, adjusted according to the actual situation of the application, 10240 is recommended
• daemon: Daemon process mode, Haproxy can be started in non-daemon process mode, it is recommended to use daemon process mode to start
• nbproc: The number of concurrent processes for load balancing. It is recommended to be equal to or 2 times the number of CPU cores of the current server
• retries: The number of retries, mainly used to check cluster nodes. If there are many nodes and the amount of concurrency is large, set it to 2 or 3 times
• option http-server-close: Actively close the http request option, it is recommended to use this option in a production environment
• timeout http-keep-alive: Long connection timeout time, set the long connection timeout time, which can be set to 10s
• timeout http-request: http request timeout time, it is recommended to set this time to 5～10s to increase the http connection release speed
• timeout client: client timeout time. If the traffic is too large and the node response is slow, you can set this time shorter, and it is recommended to set it to about 1min.