In general WebSockets don't play well with proxies and load balancers. Deploying a SockJS server behind Nginx or Apache can be a pain.
Fortunately, recent versions of HAProxy, an excellent load balancer, are capable of proxying WebSocket connections. We recommend HAProxy as a front-end load balancer.
Environmental preparation
current system version
implement:lua -v
Lua 5.3.4 Copyright (C) 1994-2017 Lua.org, PUC-Rio
Install basic commands and compile dependent environments
yum install gcc readline-devel -y
#下载,安装 Lua
mkdir /opt/app
cd /opt/app
wget https://www.lua.org/ftp/lua-5.4.4.tar.gz
tar xvf lua-5.4.4.tar.gz -C /usr/local/src/
rm /opt/app/lua-5.4.4.tar.gz
cd /usr/local/src/lua-5.4.4/
make linux test
src/lua -v
输出: Lua 5.4.4 Copyright (C) 1994-2022 Lua.org, PUC-Rio
Modify file and process handle limits
Open /etc/pam.d/loginthe file and append at the end:
#wjw_add
session required /lib64/security/pam_limits.so
In order for the limits.conf file configuration to take effect, the pam_limits.so module file must be added to the startup file
Open /etc/security/limits.confthe file and append at the end:
* soft nofile 120000
* hard nofile 120000
* soft nproc 120000
* hard nproc 120000
The preceding * indicates the current user, hard and soft indicate the limit and warning limit respectively, nofile indicates the maximum number of files, and the following number 120000 indicates that any user can open 120,000 files
To take effect without reboot execute:
ulimit -n 120000andulimit -n 120000
Open /etc/sysctl.confthe file and append at the end:
#wjw_add for HaProxy
net.ipv4.tcp_tw_reuse = 1 # 表示开启重用.允许将TIME-WAIT sockets重新用于新的TCP连接,默认为0,表示关闭;
net.ipv4.tcp_tw_recycle = 1 # 表示开启TCP连接中TIME-WAIT sockets的快速回收,默认为0,表示关闭.
net.ipv4.tcp_syncookies=1 # 表示开启SYN Cookies.当出现SYN等待队列溢出时,启用cookies来处理,可防范少量SYN攻击,默认为0,表示关闭;
net.ipv4.conf.all.rp_filter = 1 # 过滤反向路由不通的数据包
net.ipv4.ip_local_port_range = 1024 65535 # 表示用于向外连接的端口范围.缺省情况下很小:32768到61000,改为1024到65000.
net.ipv4.tcp_max_syn_backlog = 50000 # 记录的那些尚未收到客户端确认信息的连接请求的最大值,可以容纳更多等待连接的网络连接数. 对于有128M内存的系统而言,缺省值是1024,小内存的系统则是128.
net.ipv4.tcp_max_tw_buckets = 400000 #timewait的数量,默认是180000.
net.ipv4.tcp_max_orphans = 60000 # 系统中最多有多少个TCP套接字不被关联到任何一个用户文件句柄上.如果超过这个数字,孤儿连接将即刻被复位并打印出警告信息.这个限制仅仅是为了防止简单的DoS攻击,不能过分依靠它或者人为地减小这个值,更应该增加这个值(如果增加了内存之后).
net.ipv4.tcp_synack_retries = 3 # 为了打开对端的连接,内核需要发送一个SYN并附带一个回应前面一个SYN的ACK.也就是所谓三次握手中的第二次握手.这个设置决定了内核放弃连接之前发送SYN+ACK包的数量.
net.core.somaxconn = 50000 # web应用中listen函数的backlog默认会给我们内核参数的net.core.somaxconn限制到128
net.core.rmem_max = 16777216 # 系统套接字缓冲区(读), 包含三个整数值，分别是：min，default，max
net.core.wmem_max = 16777216 # 系统套接字缓冲区(写), 包含三个整数值，分别是：min，default，max
net.ipv4.tcp_rmem = 4096 87380 16777216 # TCP接收／发送缓冲区(读)
net.ipv4.tcp_wmem = 4096 65536 16777216 # TCP接收／发送缓冲区(写)
net.ipv4.tcp_no_metrics_save = 1 # 使metrics不保存每个单独的数据包
net.ipv4.tcp_moderate_rcvbuf = 1 # 表示打开了TCP内存自动调整功能
net.core.netdev_max_backlog = 50000 # 每个网络接口接收数据包的速率比内核处理这些包的速率快时,允许送到队列的数据包的最大数目.
fs.file-max=120000 # 表示系统级别的能够打开的文件句柄①的数量.是对整个系统的限制,并不是针对用户的.
Execute sysctl -pthe command action to make our changes take effect.
Compile and install HaProxy
Compilation parameters for HaProxy 2.0 and above:
yum -y install gcc openssl-devel pcre-devel systemd-devel
cd ~
wget https://www.haproxy.org/download/2.5/src/haproxy-2.5.12.tar.gz
tar xvf haproxy-2.5.12.tar.gz -C /usr/local/src
rm -y ./haproxy-2.5.12.tar.gz
cd /usr/local/src/haproxy-2.5.12/
View installation method
less INSTALL
less Makefile
Refer to the INSTALL file to compile and install
make clean
make ARCH=x86_64 TARGET=linux-glibc USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1 USE_LUA=1 LUA_INC=/usr/local/src/lua-5.4.4/src/ LUA_LIB=/usr/local/src/lua-5.4.4/src/
mkdir -p /opt/app/haproxy
make install PREFIX=/opt/app/haproxy/
tree /opt/app/haproxy/
ln -s /opt/app/haproxy/sbin/haproxy /usr/sbin/
haproxy -h
“USE_PCRE=1” : enable PCRE version 1, dynamic linking
Verify HAProxy version
# which haproxy
/sbin/haproxy
# haproxy -V
HAProxy version 2.5.12-1275e54 2023/01/24 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.12.html
Running on: Linux 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64
[root@vm172-30-64-189 haproxy-2.5.12]# haproxy -V
HAProxy version 2.5.12-1275e54 2023/01/24 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.12.html
Running on: Linux 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64
Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ]
[ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*]
-v displays version ; -vv shows known build options.
-d enters debug mode ; -db only disables background mode.
-dM[<byte>] poisons memory with <byte> (defaults to 0x50)
-V enters verbose mode (disables quiet mode)
-D goes daemon ; -C changes to <dir> before loading files.
-W master-worker mode.
-Ws master-worker mode with systemd notify support.
-q quiet mode : don't display messages
-c check mode : only check config files and exit
-cc check condition : evaluate a condition and exit
-n sets the maximum total # of connections (uses ulimit -n)
-m limits the usable amount of memory (in MB)
-N sets the default, per-proxy maximum # of connections (0)
-L set local peer name (default to hostname)
-p writes pids of all children to this file
-de disables epoll() usage even when available
-dp disables poll() usage even when available
-dS disables splice usage (broken on old kernels)
-dG disables getaddrinfo() usage
-dR disables SO_REUSEPORT usage
-dL dumps loaded object files after config checks
-dr ignores server address resolution failures
-dV disables SSL verify on servers side
-dW fails if any warning is emitted
-dD diagnostic mode : warn about suspicious configuration statements
-sf/-st [pid ]* finishes/terminates old pids.
-x <unix_socket> get listening sockets from a unix socket
-S <bind>[,<bind options>...] new master CLI
Prepare the HaProxy startup file
Create a service file for HaProxy
vim /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HaProxy Load Balancer
After=syslog.target network.target
[Service]
ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
LimitNOFILE=100000
[Install]
WantedBy=multi-user.target
Create a configuration file for HaProxy
mkdir /etc/haproxy
vi /etc/haproxy/haproxy.cfg
global
daemon #以后台形式运行harpoxy
maxconn 100000 #默认最大连接数
chroot /opt/app/haproxy #chroot运行路径
user haproxy #运行haproxy 用户
group haproxy #运行haproxy 用户组
pidfile /var/lib/haproxy/haproxy.pid #haproxy 进程PID文件
defaults
mode http #所处理的类别(7层代理http，4层代理tcp)
maxconn 100000 #最大连接数
option http-keep-alive #启用从客户端到服务器的持久连接
option forwardfor #透传客户端真实IP至后端web服务器
timeout connect 10s #haproxy与后端服务器连接超时时间，如果在同一个局域网可设置较小的时间
timeout client 60s #定义客户端与haproxy连接后，数据传输完毕，不再有数据传输，即非活动连接的超时时间
timeout server 60s #定义haproxy与上游服务器非活动连接的超时时间
listen stats
mode http #所处理的类别(7层代理http，4层代理tcp)
bind 0.0.0.0:9000
log global #开启日志功能，默认不记录日志
stats enable
stats refresh 30s
stats show-node
stats auth admin:admin
stats uri /haproxy_stats
stats realm HaProxy-Statistics
frontend http_frontend
bind *:8443
maxconn 10000 #最大连接数
log global #开启日志功能，默认不记录日志
default_backend http_backend
backend http_backend
balance roundrobin
#weigth：设置后端真实服务器的权重，默认为1，最大值为256，设置为0表示不参与负载均衡。
#cookie 为指定的后端服务器设定cookie值，此外指定的值将在请求入站时被检查，第一次为此值挑选的后端服务器将在后续的请求中一直被选中，其目的在于实现持久连接的功能。
#check：表示启用对此后端服务器执行健康检查。
#maxconn：设定每个backend中server进程可接受的最大并发连接数，此选项等同于linux命令选项”ulimit -n”。
server websrv1 192.168.10.11:8080 maxconn 10000 weight 10 cookie websrv1 check
server websrv2 192.168.10.12:8080 maxconn 10000 weight 10 cookie websrv2 check
Enable the HaProxy service
systemctl daemon-reload
Reload the configuration file of a certain service. If a new service is installed, it belongs to systemctl management. If the service program configuration file of the new service takes effect, it needs to be reloaded.
Start HaProxy
#准备socket文件目录
mkdir /var/lib/haproxy
#设置用户和目录权限
useradd -r -s /sbin/nologin -d /var/lib/haproxy haproxy
systemctl enable --now haproxy
Verify HaProxy status
systemctl status haproxy
输出:
● haproxy.service - HaProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2023-02-08 18:35:20 CST; 28s ago
Process: 260383 ExecStartPre=/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q (code=exited, status=0/SUCCESS)
Main PID: 260386 (haproxy)
Tasks: 5 (limit: 49524)
Memory: 19.4M
CGroup: /system.slice/haproxy.service
├─260386 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
└─260388 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /var/lib/haproxy/haproxy.pid
Feb 08 18:35:20 vm172-30-64-189 systemd[1]: Starting HaProxy Load Balancer...
Feb 08 18:35:20 vm172-30-64-189 systemd[1]: Started HaProxy Load Balancer.
Feb 08 18:35:20 vm172-30-64-189 haproxy[260386]: [NOTICE] (260386) : New worker (260388) forked
Feb 08 18:35:20 vm172-30-64-189 haproxy[260386]: [NOTICE] (260386) : Loading success.
View haproxy status page
浏览器访问：http://10.0.0.7:9000/haproxy_stats 用户名：admin 密码：admin
HaProxy service management
# 检查配置文件合法性
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c
#显示状态
systemctl status haproxy.service
#启动
systemctl start haproxy.service
#停止
systemctl stop haproxy.service
#重启
systemctl restart haproxy.service
#设置开机启动
systemctl enable haproxy.service
#关闭开机启动
systemctl disable haproxy.service
#查看开机自启是否成功
systemctl is-enabled haproxy.service
HaProxy log configuration
Author's note: HaProxy itself does not record client access logs. In addition, in order to reduce server load, HaProxy generally does not record logs in production.
Since haproxy does not have a separate log, it is necessary to create a monitoring log to facilitate troubleshooting
Modify the haproxy configuration file
vim /etc/haproxy/haproxy.cfg
#在global配置项定义：
log 127.0.0.1 local2 info
# 在 listen 或者 frontend 配置项定义：
log global #开启日志功能，默认不记录日志
#然后重启haproxy服务:
systemctl restart haproxy
Modify the rsyslog configuration file
vim /etc/rsyslog.conf
#放开这两行注释
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
......
local2.* /var/log/haproxy.log
......
#然后重启rsyslog服务:
systemctl restart rsyslog
Verify haproxy logs
Visit the "http://10.0.0.7:9000/haproxy_stats" page and verify that logs are generated.
or "wget http://127.0.0.1:9000/haproxy_stats"
Use the following command to view the haproxy log:tail -f /var/log/haproxy.log
appendix:
Detailed HaProxy configuration
proxies configuration
defaults [<name>] #默认配置项，针对以下的frontend、backend和listen生效，可以多个name也可以没有name
frontend <name> #前端servername，类似于Nginx的一个虚拟主机 server和LVS服务集群。
backend <name> #后端服务器组，等于nginx的upstream和LVS中的RS服务器
listen <name> #将frontend和backend合并在一起配置，相对于frontend和backend配置更简洁，生产常用
defaults
defaults 配置参数，不需要改动：
option redispatch #当server Id对应的服务器挂掉后，强制定向到其他健康的服务器，重新派发
option abortonclose #当服务器负载很高时，自动结束掉当前队列处理比较久的连接，针对业务情况选择开启
option http-keep-alive #开启与客户端的会话保持
option forwardfor #透传客户端真实IP至后端web服务器
mode http|tcp #设置默认工作类型,使用TCP服务器性能更好，减少压力
timeout http-keep-alive 120s #session 会话保持超时时间，此时间段内会转发到相同的后端服务器
timeout connect 120s #客户端请求从haproxy到后端server最长连接等待时间(TCP连接之前)，默认单位ms
timeout server 600s #客户端请求从haproxy到后端服务端的请求处理超时时长(TCP连接之后)，默认单位ms，如果超时，会出现502错误，此值建议设置较大些，访止502错误
timeout client 600s #设置haproxy与客户端的最长非活动时间，默认单位ms，建议和timeout server相同
timeout check 5s #对后端服务器的默认检测超时时间
default-server inter 1000 weight 3 #指定后端服务器的默认设置
listen (simplified configuration of fronted & backend)
使用listen替换 frontend和backend的配置方式，可以简化设置，通常只用于TCP协议的应用
listen WEB_PORT_80
bind 10.0.0.7:80
mode http
option forwardfor #透传客户端真实IP至后端web服务器
server web1 10.0.0.18:80 check inter 3000 fall 2 rise 5
server web2 10.0.0.28:80 check inter 3000 fall 2 rise 5
fronted
frontend 配置参数：
bind: #指定haproxy的监听地址，可以同时监听多个IP或端口，可同时用于listen字段中
#格式
bind [<address>]:<port_range> [, ...] [param*]
#注意：如果需要绑定在非本机的IP，需要开启内核参数：net.ipv4.ip_nonlocal_bind=1
backlog <backlog> #针对所有server配置,当前端服务器的连接数达到上限后的后援队列长度，注意：不支持backend
范例：
listen http_proxy #监听http的多个IP的多个端口和sock文件
bind :80,:443,:8801-8810
bind 10.0.0.1:10080,10.0.0.1:10443
bind /var/run/ssl-frontend.sock user root mode 600 accept-proxy
listen http_https_proxy #https监听
bind :80
bind :443 ssl crt /etc/haproxy/site.pem #公钥和私钥公共文件
listen http_https_proxy_explicit #监听ipv6、ipv4和unix sock文件
bind ipv6@:80
bind ipv4@public_ssl:443 ssl crt /etc/haproxy/site.pem
bind [email protected] user root mode 600 accept-proxy
listen external_bind_app1 #监听file descriptor
bind "fd@${FD_APP1}"
生产范例：
frontend magedu_web_port #可以采用后面形式命名：业务-服务-端口号
bind :80,:8080
bind 10.0.0.7:10080,:8801-8810,10.0.0.17:9001-9010
mode http|tcp #指定负载协议类型
use_backend <backend_name> #调用的后端服务器组名称
backend
定义一组后端服务器，backend服务器将被frontend进行调用。
注意: backend 的名称必须唯一,并且必须在listen或frontend中事先定义才可以使用,否则服务无法启动
mode http|tcp #指定负载协议类型,和对应的frontend必须一致
option #配置选项
server #定义后端real server,必须指定IP和端口
注意：option后面加 httpchk，smtpchk,mysql-check,pgsql-check，ssl-hello-chk方法，可用于实现更多应用层检测功能。
server 配置
#针对一个server配置
check #对指定real进行监控状态检查，默认不开启
addr <IP> #可指定的健康状态监测IP，可以是专门的数据网段，减少业务网络的流量
port <num> #指定的健康状态监测端口
inter <num> #健康状态检查间隔时间，默认2000 ms
fall <num> #后端服务器从线上转为线下的检查的连续失效次数，默认为3
rise <num> #后端服务器从下线恢复上线的检查的连续有效次数，默认为2
weight <weight> #权重，默认为1
backup #将后端服务器标记为备份状态，只有所有服务器宕机时提供服务，类似sorry server
disabled #将后端服务器标记为不可用状态
redir http://www.baiud.com #临时重定向（302）至其它URL，只适用于http模式
maxconn <maxconn> #当前后端server的最大并发连接数
<<<<<< [完] >>>>>>