introduction

As the world's leading open source distributed MQTT Broker, EMQX introduced MQTT over QUIC in version 5.0, combining the advantages of the MQTT protocol with the features of QUIC. By making full use of the low connection overhead and multiplexing features of the QUIC protocol, MQTT over QUIC provides a very promising solution for users in weak network environments and irregular networks. It is able to cope with connection loss and slow connection establishment in IoT scenarios such as connected vehicles operating in harsh environments such as mountains or tunnels. With the development of cloud native technology, more and more users choose to deploy EMQX clusters on Kubernetes to enjoy the advantages of rapid creation and convenient management. This article will introduce how to deploy an EMQX cluster on Kubernetes and enable the MQTT over QUIC function.

Expose EMQX service

When deploying EMQX on Kubernetes, you can use LoadBalanceror NodePortto expose EMQX services to clients outside the cluster.

LoadBalancerThe method relies on the load balancer provided by the cloud service provider to provide services. Currently, the load balancer of the cloud service provider does not support the address migration feature of QUIC.
NodePortThe method relies on the kube-proxy component of Kubernetes to forward external requests, it can seamlessly connect to EMQX services, and supports the QUIC address migration feature.

In the Internet of Vehicles scenario, the address of the vehicle end may change frequently, so the address migration feature of QUIC is very important. Therefore, when deploying EMQX 5.0 with the MQTT over QUIC function on Kubernetes, we recommend that you choose to NodePortexpose the service externally.

Below, we will introduce the specific steps to deploy EMQX 5.0 on Kubernetes and enable MQTT over QUIC. At the same time, we will NodePortexpose the service and verify the address migration function of QUIC in a way.

prerequisite

Before deploying EMQX 5.0 to Kubernetes, please ensure the following requirements are met:

Kubernetes version >= 1.27

When the K8s version is lower than 1.27, UDP packets are discarded due to the bug of kube-proxy IPVS break UDP NodePort Services . At present, this bug has been fixed in K8s version 1.27. For details, please refer to: Syncing IPVS conntrack cleaning with IPTables . If your K8s cluster version is lower than 1.27, it is recommended to use kube-proxy in iptables mode.
Helm version >= 3

Install EMQX Operator

Install and start cert-manager.

cert-managerVersion needs to be equal to or higher than 1.1.6. If it is already installed and started cert-manager, please skip this step.
```
$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm upgrade --install cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true
```
You can also refer to the cert-manager installation guide for installation.

Use Helm to install EMQX Operator.

$ helm repo add emqx https://repos.emqx.io/charts
$ helm repo update
$ helm install emqx-operator emqx/ emqx-operator --namespace emqx-operator-system --create-namespace

Wait for EMQX Operator to be ready.

$ kubectl wait --for=condition=Ready pods -l "control-plane=controller-manager" -n emqx-operator-system

# 如果您得到类似以下的输出结果，说明 emqx-operator 已经就绪：
pod/emqx-operator-controller-manager-57bd7b8bd4-h2mcr condition met

Deploy EMQX 5.0 and enable MQTT over QUIC

Save the following content as a YAML file and use kubectl applythe command to deploy.

apiVersion: apps.emqx.io/v2alpha1
kind: EMQX
metadata:
name: emqx
spec:
image: emqx:5.0
bootstrapConfig: |
  listeners.quic.default {
    enabled = true
    bind = "0.0.0.0:14567"
    max_connections = 1024000
    keyfile = "/opt/emqx/etc/certs/key.pem"
    certfile = "/opt/emqx/etc/certs/cert.pem"
  }
coreTemplate:
  spec:
    replicas: 3
replicantTemplate:
  spec:
    replicas: 3
listenersServiceTemplate:
  spec:
    type: NodePort
    ports:
      - name: quic-default
        protocol: UDP
        port: 14567
        targetPort: 14567

listeners.quic.defaultIndicates that the QUIC listener is enabled and the UDP 14567port is bound.

Wait for the EMQX cluster to be ready. You can kubectl getview the status of the EMQX cluster through the command, please make sure STATUSit is Running. This may take some time.
```
$ kubectl get emqx
NAME   IMAGE      STATUS    AGE
emqx   emqx:5.0   Running   10m
```

Obtain the listener service of the EMQX cluster.

EMQX Operator will create two EMQX Service resources, namely emqx-dashboardand emqx-listeners, which are used for EMQX console and EMQX listening port.

$ kubectl get service emqx-listeners
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
emqx-listeners   NodePort   192.168.50.64   <none>        14567:30171/UDP,1883:32633/TCP   2m1s

You can see that the QUIC listener is enabled in the service.

Test QUIC with eMQTT-Bench

eMQTT-Bench is a lightweight MQTT 5.0 benchmark tool written in Erlang. You can download and install eMQTT-Bench that supports the QUIC protocol for your platform from the eMQTT-Bench release page .

Use the QUIC protocol to establish a connection and --quicsubscribe with the specified option. Here are 10 clients subscribed to t/testthe topic.
```
$ ./emqtt_bench sub --quic -h ${node_ip} -p ${node_port} -t t/test -c 10
```
Open another terminal, connect using the QUIC protocol and perform a release test.
```
$ ./emqtt_bench pub --quic -h ${node_ip} -p ${node_port} -t t/test -c 1
```
At this point, you can see the message subscription publishing rate of the subscriber and the publisher from the output log of the command line.
Perform an address migration test.

We switch the client network at the time point marked by the arrow in the figure, and observe the situation of the EMQX cluster sending and receiving messages:

As can be seen from the above figure, when the client network changes, QUIC has no effect on the receiving and sending of messages. At the same time, there is no abnormality in the client publishing and subscribing messages, as shown in the following figure:

Challenges of using QUIC on Kubernetes

At present, the main problem of using the QUIC protocol on Kubernetes is that the load balancer provided by the cloud service provider does not fully support the QUIC protocol, such as not supporting the IETF QUIC protocol and the QUIC address migration feature.

epilogue

The above is the whole process of using EMQX 5.0 on Kubernetes to experience MQTT over QUIC. It can be seen that deploying EMQX 5.0 on Kubernetes is very simple and only needs a YAML file to complete. After enabling MQTT over QUIC, your device can communicate with the EMQX cluster based on the QUIC protocol, making full use of its advantages in IoT message transmission.

Copyright statement: This article is original by EMQ, please indicate the source for reprinting.
Original link: https://www.emqx.com/zh/blog/enabling-mqtt-over-quic-on-kubernetes-with-emqx-5-0

Experience the MQTT over QUIC feature of EMQX 5.0 on Kubernetes