Is it possible to use EMQ to carry millions of user connections? The official reply is that the configuration of 8 cores and 32G can carry links of 160W devices, so what is the performance? You can only know if you try it yourself. Let's have a look at the true face of EMQ by tuning the system configuration and stress testing EMQ.
attach:
Meow's blog: w-blog.cn EMQ official address: http://emqtt.com/ EMQ Chinese documentation: http://emqtt.com/docs/v2/guide.html
1.Liunx and Erlang virtual machine tuning
Linux system parameter optimization
Modify the number of files that can be opened by all processes in the system
sysctl -w fs.file-max=2097152
sysctl -w fs.nr_open=2097152
> vi /etc/sysctl.conf
fs.file-max = 2097152
fs.nr_open = 2097152
Set the maximum number of file handles for the service
vim /etc/systemd/system.conf
DefaultLimitNOFILE=1048576
Persistence settings allow the user/process to open the number of file handles:
ulimit -n 1048576
> vim /etc/security/limits.conf
* soft nofile 1048576
* hard nofile 1048576
The '*' sign can be used to modify the limit of all users; soft or hard specifies whether to modify the soft limit or the hard limit; 10240 specifies the new limit value to be modified, that is, the maximum number of open files (please note that the soft limit value should be less than or equal to the hard limit).
TCP stack network parameters
> vi /etc/sysctl.conf
### backlog - Socket 监听队列长度:
net.core.somaxconn=32768
net.ipv4.tcp_max_syn_backlog=16384
net.core.netdev_max_backlog=16384
## 可用知名端口范围:
net.ipv4.ip_local_port_range='1000 65535'
## TCP Socket 读写 Buffer 设置:
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
#sysctl -w net.ipv4.tcp_mem='16777216 16777216 16777216'
net.ipv4.tcp_rmem='1024 4096 16777216'
net.ipv4.tcp_wmem='1024 4096 16777216'
## TCP 连接追踪设置:
net.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_max=1000000
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
## FIN-WAIT-2 Socket 超时设置:
net.ipv4.tcp_fin_timeout = 15
## TIME-WAIT Socket 最大数量、回收与重用设置:
net.ipv4.tcp_max_tw_buckets=1048576
# 注意: 不建议开启該设置，NAT模式下可能引起连接RST
# net.ipv4.tcp_tw_recycle = 1
# net.ipv4.tcp_tw_reuse = 1
Erlang virtual machine parameters
> vim /usr/local/emqttd/etc/emq.conf
## Erlang Process Limit
node.process_limit = 2097152
## Sets the maximum number of simultaneously existing ports for this system
node.max_ports = 1048576
## EMQ 最大允许连接数
## Size of acceptor pool
listener.tcp.external.acceptors = 64
## Maximum number of concurrent clients(以1G内存比5W进行配置)
listener.tcp.external.max_clients = 1000000
After restarting emq, you can see the following display on the Dashboard:
2. Stress test program EMQ
The stress test requires an environment of erlang R17 or above (the default yum is the R16 version, if yum can be installed to a new version, please ignore it)
## 依赖
yum -y install ncurses-devel openssl-devel unixODBC-devel gcc-c++
mkdir -p /app/install $$ cd /app/install/
wget http://erlang.org/download/otp_src_19.0.tar.gz
tar -xvzf otp_src_19.0.tar.gz
cd otp_src_19.0
./configure --prefix=/usr/local/erlang --with-ssl -enable-threads -enable-smmp-support -enable-kernel-poll --enable-hipe --without-javac
make && make install
Configure erl environment variables
vim /etc/profile
# erlang
export ERLPATH=/usr/local/erlang
export PATH=$ERLPATH/bin:$PATH
source /etc/profile
Install pressure measurement software
yum -y install git
cd /app/install
git clone https://github.com/emqtt/emqtt_benchmark.git
cd emqtt_benchmark
## 调整系统参数并且开始压测
sysctl -w net.ipv4.ip_local_port_range="500 65535"
echo 1000000 > /proc/sys/fs/nr_open
ulimit -n 1000000
./emqtt_bench_sub -h 192.168.2.111 -c 32219 -i 1 -t bench /％i -q 2
Attached is the author's stress test chart: The author used 14 servers with 1 core and 1G to stress the EMQ server with 2 cores and 8G, and got a stable link peak of 44W. It can be known that the best ratio is 1G memory corresponding to 6W device connection, and the official The connection number of the given 32G memory 160W devices is very close
3 Summary
Through the stress test after system tuning, we have basically obtained the same data as the official data. It can be seen that the number of connections that EMQ can carry is really amazing. It is called a million-level message service. After getting this conclusion, our next step is to start the cluster. Exploration of permission restrictions, we will see you soon...
Note: The author has limited ability and I hope that everyone can point out the wrong places, and I hope to communicate more!