Haproxy+Keepalived搭建高可用负载均衡的EMQ集群

一、整体架构

1.1、架构图

1.2、架构服务器

服务器IP	部署业务	作用
192.168.0.41	EMQTTD	EMQ集群
192.168.0.75	EMQTTD	EMQ集群
192.168.0.41	haproxy、keepalived	HA和LB
192.168.0.75	haproxy、keepalived	HA和LB

1.3、架构说明

Haproxy和Keepalived以及EMQ都放在同一台机器，每一个机器上都有一个MEQ、Haproxy、Keepalived，其实可以把Haproxy、Keepalived单独弄两台机器，每台机器上放一个Haproxy、Keepalived，切记Haproxy和Keepalived必须放在同一台机器上，然后Haproxy负载到不同的EMQ机器上即可，但是我用来部署的机器没那么多，只能放在一起了，有条件的朋友可以自己试试分开的，其实都一样。

环境：Cetos7，EMQ2.3.11

1.4、操作系统配置

1.4.1、两台机器都需要进行一下配置，如果将EMQ和Haproxy、Keepalved分开机器放，那么每个机器都要配置。

关闭防火墙，关闭开机启动：
systemctl stop firewalld.service #关闭防火墙
systemctl disable firewalld.service #防火墙开机不启动
关闭selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
重启服务器
shutdown -r now

1.4.2、修改操作系统内核参数

cat << EOF >> /etc/sysctl.conf
fs.file-max=2097152 
fs.nr_open=2097152
net.core.somaxconn=32768
net.ipv4.tcp_max_syn_backlog=16384
net.core.netdev_max_backlog=16384
net.ipv4.ip_local_port_range=1000 65535
net.core.rmem_default=262144
net.core.wmem_default=262144
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.core.optmem_max=16777216
net.ipv4.tcp_rmem=1024 4096 16777216
net.ipv4.tcp_wmem=1024 4096 16777216
net.ipv4.tcp_max_tw_buckets=1048576
net.ipv4.tcp_fin_timeout = 15
EOF

cat << EOF >>/etc/security/limits.conf
      soft   nofile      1048576
      hard   nofile      1048576
EOF

echo DefaultLimitNOFILE=1048576 >>/etc/systemd/system.conf

修改完毕通过sysctl –p 生效

二、EMQ集群的部署

2.1、EMQ的安装

我之前写过EMQ安装的文章，分别使用通用包、源码编译安装，这里就不再讲了，自己去看看。

传送门：Linux下用EMQ通用包安装EMQ、Linux下用源码编译安装EMQ

2.2、修改EMQ配置文件，设置最大连接限制等参数

进入emqx-rel-2.3.11/_rel/emqttd/etc/emq.conf中

sed -i 's%^node\.process_limit = .*%node.process_limit = 2097152%g' emq.conf
sed -i 's%^node\.max_ports = .*%node.max_ports = 1048576%g' emqttd/emq.conf
sed -i 's%^listener\.tcp.external\.acceptors = .*%listener.tcp.external.acceptors = 64%g' emq.conf
sed -i 's%^listener\.tcp\.external\.max_clients = .*%listener.tcp.external.max_clients = 1000000%g' emq.conf

2.3、EMQ集群的搭建

2.3.1、节点设置，修改emqx-rel-2.3.11/_rel/emqttd/etc/emq.conf文件中的节点名以及集群名

一共两种方式：进入emqx-rel-2.3.11/_rel/emqttd/etc/emq.conf

通过vim修改，cluster.name = emqcl，node.name = [email protected]；（192.168.0.41）

cluster.name = emqcl，node.name = [email protected]；（192.168.0.75）

或者通过命令直接修改值

192.168.0.41：

sed -i 's%^node\.name = .*%node\.name = [email protected]%g' emq.conf

sed -i 's%^cluster\.name = .*%cluster.name = emqcl%g' emq.conf

192.168.0.75：

sed -i 's%^node\.name = .*%node\.name = [email protected]%g' emq.conf

sed -i 's%^cluster\.name = .*%cluster.name = emqcl%g' emq.conf

2.3.2、加入集群

在192.168.0.41的emqx-rel-2.3.11/_rel/emqttd/bin下运行/emqttd_ctl cluster join [email protected]

或者在192.168.0.75的emqx-rel-2.3.11/_rel/emqttd/bin下运行/emqttd_ctl cluster join [email protected]

注：节点启动加入集群后，节点名称不能变更。若更改了节点名称或者集群名，重启emq服务，./emqttd_ctl console,

./emqttd_ctl start。若重启服务还是无效，则重启服务器，shutdown -r now

2.3.3、查看集群状态

在任何一个节点的emqx-rel-2.3.11/_rel/emqttd/bin下运行/emqttd_ctl cluster status

2.3.4、退出集群，有两种方式:

leave: 本节点退出集群
remove: 从集群删除其他节点

在192.168.0.41的emqx-rel-2.3.11/_rel/emqttd/bin下运行./emqttd_ctl cluster leave

或者在192.168.0.41的emqx-rel-2.3.11/_rel/emqttd/bin下运行./emqttd_ctl cluster remove [email protected]

三、Haproxy的部署

3.1、安装Haproxy，yum -y install haproxy

3.2、修改haproxy.cfg文件，两台机器的配置文件一样

安装完haproxy以后，默认生成/etc/haproxy/haproxy.cfg，因为本身生成的，有其自带的东西，最好删除了，再运行以下命令重建。

[root@jmeter-1 haproxy]# cat <<EOF>>/etc/haproxy/haproxy.cfg
> #---------------------------------------------------------------------
> # Example configuration for a possible web application.  See the
> # full configuration options online.
> #
> #   http://haproxy.1wt.eu/download/1.4/doc/configuration.txt
> #
> #---------------------------------------------------------------------
> 
> #---------------------------------------------------------------------
> # Global settings
> #---------------------------------------------------------------------
> global
>     # to have these messages end up in /var/log/haproxy.log you will
>     # need to:
>     #
>     # 1) configure syslog to accept network log events.  This is done
>     #    by adding the '-r' option to the SYSLOGD_OPTIONS in
>     #    /etc/sysconfig/syslog
>     #
>     # 2) configure local2 events to go to the /var/log/haproxy.log
>     #   file. A line like the following can be added to
>     #   /etc/sysconfig/syslog
>     #
>     # local2.*                       /var/log/haproxy.log
>     #
>     log         127.0.0.1 local2
> 
>     # chroot      /usr/sbin/haproxy
>     pidfile     /var/run/haproxy.pid
>     maxconn     1000000
>     user        root
>     group       root
>     daemon
> 
>     # turn on stats unix socket
>     # stats socket /var/lib/haproxy/stats
>     # stats socket /usr/sbin/haproxy/stats
> 
> #---------------------------------------------------------------------
> # common defaults that all the 'listen' and 'backend' sections will
> # use if not designated in their block
> #---------------------------------------------------------------------
> defaults
>     log                     global
>     option                  dontlognull
>     option http-server-close
>     # option forwardfor
>     retries                 3
>     timeout http-request    10s
>     timeout queue           1m
>     timeout connect         60s
>     timeout client          2m
>     timeout server          2m
>     timeout http-keep-alive 10s
>     timeout check           10s
> 
> frontend emqtt-front
>     bind *:1885
>     maxconn     1000000
>     mode tcp
>     default_backend emqtt-backend
> 
> backend emqtt-backend
>     balance roundrobin
>     # balance source
>     server emq-1 192.168.0.41:1883 check inter 100000 fall 2 rise 5 weight 1
>     server emq-2 192.168.0.75:1883 check inter 100000 fall 2 rise 5 weight 1
>     # source 0.0.0.0 usesrc clientip
> 
> frontend emqtt-pub-front
>     bind *:8085
>     maxconn     1000000
>     mode tcp
>     default_backend emqtt-pub-backend
> 
> backend emqtt-pub-backend
>     balance roundrobin
>     # balance source
>     server emq-1 192.168.0.41:8083 check inter 100000 fall 2 rise 5 weight 1
>     server emq-2 192.168.0.75:8083 check inter 100000 fall 2 rise 5 weight 1
> 
> frontend emqtt-admin-front
>     bind *:18085
>     mode http
>     default_backend emqtt-admin-backend
> 
> backend emqtt-admin-backend
>     mode http
>     balance roundrobin
>     server emq-1 192.168.0.41:18083 check
>     server emq-2 192.168.0.75:18083 check
> listen admin_stats
>         stats   enable
>         bind    *:8082
>         mode    http 
>         option  httplog
>         log     global
>         maxconn 10
>         stats   refresh 30s
>         stats   uri /admin
>         stats   realm haproxy
>         stats   auth admin:admin
>         stats   hide-version
>         stats   admin if TRUE
> EOF

注：这里有个问题，如果你的haproxy、keepalived和emq是放在同一个机器上的，haproxy的配置文件中bind绑定的端口号不要跟EMQ的一样，不然haproxy启动报错。如果不是同一台机器上的，那就尽量端口一样。

3.3、启动haproxy，systemctl start haproxy

设置开机启动：chkconfig haproxy on

3.4、查看haproxy的状态systemctl status haproxy，若出现以下则代表启动成功

3.5、优化Haproxy服务器

echo session required /usr/lib64/security/pam_limits.so >>/etc/pam.d/login

cat << EOF >> /etc/sysctl.conf
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_syncookies = 1
EOF

修改完毕通过sysctl –p 生效

四、Keepalived的部署

4.1安装keepalived， yum -y install keepalived

4.2修改keepalived.cfg文件，两台机器的配置文件有点区别，指定主备

安装完keepalived以后，默认生成/etc/keepalived/keepalived.cfg，删除该文件，再运行以下命令重建。

主的keepalived.cfg：