Getting Started Guide for MQTT Protocol Client Development

introduction

Hello everyone, this is Anyin.

Some time ago, because my work involved dealing with hardware devices, I did some MQTT-related work. Today, I will also do a simple sharing here.

basic concepts

Before doing related development work, we need to understand what MQTT is.

MQTT is a lightweight publish-subscribe mode messaging protocol, specially designed for IoT applications in low bandwidth and unstable network environments.

• MQTT official website
• MQTT V3.1.1 protocol specification

The MQTT protocol has the following characteristics:

• Open message protocol, simple and easy to implement
• Publish-subscribe model, one-to-many message publishing
• TCP/IP based network connection
• 1-byte fixed header, 2-byte heartbeat message, compact message structure
• Message QoS support, reliable transmission guarantee

The main application scenarios of MQTT:

• IoT M2M communication, IoT big data collection
• Android message push, WEB message push
• Mobile instant messaging, such as Facebook Messenger
• Smart hardware, smart furniture, smart appliances
• Vehicle networking communication, electric station pile collection
• Smart city, telemedicine, distance education
• Industry markets such as power, oil and energy

For more details, please check the official website. It will not be repeated here.

For the installation of the MQTT server, we use it here EMQX, its official website address: www.emqx.io/zh

Implement an MQTT client

When our EMQserver is installed, we can encode our MQTT client to receive messages from the device or send messages to the device. The whole process is 异步the same.

1. pom.xmladd dependencies

        <dependency>
            <groupId>org.eclipse.paho</groupId>
            <artifactId>org.eclipse.paho.client.mqttv3</artifactId>
            <version>${mqtt.version}</version>
        </dependency>
复制代码

2. Package MqttClientinstance

In step 1, we rely on a third-party library of mqtt. In order to prevent other third-party libraries from being replaced in the future, we need to MqttClientmake a simple package for it.

Add a MQTTClientclass.

    /**
     * 实例化MQTT 实例
     * @param properties 配置信息
     * @param factory 扩展点工厂
     */
    public MQTTClient(MQTTProperties properties,
                      IExtensionHandlerFactory factory,
                      List<SubscribeTopic> subscribeTopics) throws MqttException {
        if(CollectionUtils.isEmpty(subscribeTopics)){
            throw new CommonBusinessException("-1","订阅列表不能为空");
        }
        this.subscribeTopics = subscribeTopics;
        this.properties = properties;
        this.factory = factory;
        this.clientId = "SERVER_" + RandomUtil.randomString(8);
        this.init();
    }
复制代码

• MQTTPropertiesBe familiar with the related configuration of MQTT
• IExtensionHandlerFactoryThe extension point factory component needs to perform business processing according to different instructions when receiving messages, so this component is required
• List<SubscribeTopic>The client needs to subscribe to the topic list on the device side

Next, let's look at the initmethod.

    /**
     * 初始化MQTT Client 实例
     */
    public void init() throws MqttException {
        String broker = "tcp://" + properties.getHost() + ":" + properties.getPort();
        MemoryPersistence persistence = new MemoryPersistence();
        try {
            if(client == null){
                client = new MqttClient(broker, clientId, persistence);
            }
            MqttConnectOptions connOpts = this.getOptions();
            if (client.isConnected()) {
                client.disconnect();
            }
            client.connect(connOpts);
            client.setCallback(new MQTTSubscribe(this, factory, subscribeTopics));
            // 订阅路径
            client.subscribe(this.getSubscribeTopicList());
        }catch (MqttException ex) {
            log.error("mqtt服务初始化失败: {}", ex.getMessage(), ex);
            throw ex;
        }
        log.info("mqtt服务连接成功");
    }
复制代码

这里主要处理了客户端实例连接服务器的一些操作，主要有设置参数connOpts、设置接收消息的回调setCallback,设置订阅设备端的消息主题subscribe。

这里有地方需要特别注意下，在进行连接的时候之前，做了一个client.isConnected()的判断，如果连接的状态，则需要手动的断开连接client.disconnect()。这里主要是为了做重连的时候，能够确保客户端是断开连接的状态，然后再进行重连。

客户端连接的逻辑处理了，我们还需要处理下发送消息的逻辑，简单的封装下即可。

    /**
     * 发送消息
     * @param path path
     * @param deviceId 设备ID
     * @param content 发送内容
     */
    public void publish(String path, String deviceId, byte[] content){
        try {
            MqttMessage message = new MqttMessage(content);
            message.setQos(properties.getQos());
            String topic = path + deviceId;
            client.publish( topic, message);
        }catch (Exception ex){
            log.error("mqtt服务发送消息失败： deviceId: {}  {}",deviceId, ex.getMessage(), ex);
        }
    }
复制代码

3. 处理订阅的消息

基本的客户端实例化我们已经处理完了，接着需要处理上行的消息（就是订阅的消息）。

对于不同厂商上来的业务消息可能不一样，有可能是MQTT协议包含着JSON的字符串的业务数据，也有可能是MQTT协议包含的是二进制的私有协议。

为了抽象上行的消息，我们定义了2个接口，分别抽象上行消息的整包对象和上行消息的某个指令。上行消息的整包对象就是从订阅接口返回的完整的byte[]数据包；而上行消息的某个指令是指在这个完整的数据包内肯定会有某个字段指明本次消息是属于什么业务的，可能是心跳、可能是状态等等。

分别新增一个MQTTProtocol和Cmd类。

@Data
public abstract class MQTTProtocol {
    /**
     * 设备ID
     */
    private String deviceId;

    /**
     * 消息唯一序号
     */
    private String msgId;
    /** 
     *  具体某个业务的指令
     */
    private Cmd cmd;
}

public interface Cmd {
    /**
     * 指令类型，上行的指令或者是下行的指令
     */
    CmdTypeEnum getCmdType();
    /**
     * 指令发送的目标topic
     */
    Topic getTopic();
}
复制代码

接着，我们再新增一个协议的处理器接口:MQTTProtocolHandler

public interface MQTTProtocolHandler<T extends MQTTProtocol> extends IExtensionHandler<BusinessType> {

    String getDeviceId(String topic, byte[] payload);
    /**
     * 解码
     * @param payload 原始数据
     * @return 协议
     */
    T decode(byte[] payload);

    /**
     * 校验
     * @param protocol 解析出来的基础协议
     * @param payload 原始数据
     * @return true 通过  false 不通过
     */
    boolean check(T protocol, byte[] payload);

    /**
     * 编码
     * @param protocol 协议
     * @param data 业务数据
     * @return 编码数据
     */
    byte[] encode(T protocol, byte[] data);

    /**
     * 业务处理
     * @param protocol 协议
     */
    byte[] handle(T protocol);

    /**
     * 错误响应
     * @param protocol 协议
     */
    byte[] error(T protocol);
}
复制代码

这个接口把整个消息的处理过程分为5个步骤：解码、校验、编码、业务处理、错误响应。该接口是一个扩展点，扩展点的枚举类是：BusinessType，它表示业务类型，即使不同的业务，可能会不同的编解码和处理规则。例如：JSON的数据和二进制的私有协议，它们的编解码就不一样。

然后，我们再看看当接收到消息的时候，我们如何使用这个扩展点进行业务逻辑处理。

@Override
    public void messageArrived(String subscribeTopic, MqttMessage message) throws Exception {
        try {
            // 根据topic解析不同的业务类型
            BusinessType businessType = this.matchBusinessTypeBySubscribeTopic(subscribeTopic);
            // 根据业务类型拿到具体的协议处理器
            MQTTProtocolHandler protocolHandler = extensionHandlerFactory.getExtensionHandler(businessType, MQTTProtocolHandler.class);
           // 获取设备ID
            String deviceId = protocolHandler.getDeviceId(subscribeTopic, message.getPayload());
            // 整包协议解码
            MQTTProtocol protocol = protocolHandler.decode(message.getPayload());
            if (protocol == null) {
                log.error("协议解析异常，无法进行应答");
                return;
            }
            // 指令
            Cmd cmd = protocol.getCmd();
            if(cmd == null){
                log.error("解析后指令为空，无法进行应答");
                return;
            }
            // 设置基础信息
            protocol.setMsgId(String.valueOf(message.getId()));
            protocol.setDeviceId(deviceId);

            // 校验
            boolean success = protocolHandler.check(protocol, message.getPayload());
            if(!success){
                this.errorHandle(protocolHandler, protocol, cmd.getTopic());
                return;
            }

            try {
                // 业务处理
                byte[] result = protocolHandler.handle(protocol);

                // 应答
                if(CmdTypeEnum.DOWN == cmd.getCmdType()){
                    log.info("下行消息，无需应答");
                    return;
                }
                Topic topic = cmd.getTopic();
                if(topic == null){
                    log.error("上行消息的发布Topic为空，无需进行应答");
                    return;
                }
                // 编码后进行应答
                byte[] content = protocolHandler.encode(protocol, result);
                client.publish(topic.getTopic(), deviceId, content);
            } catch (Exception ex) {
                log.error("业务逻辑处理异常: {}, 原始数据：{}", ex.getMessage(),  ByteUtil.byte2Str(message.getPayload()), ex);
                this.errorHandle(protocolHandler, protocol, cmd.getTopic());
            }
        }catch (Exception ex){
            log.error("unknown error: {}, 原始数据：{}", ex.getMessage(),  ByteUtil.byte2Str(message.getPayload()), ex);
        }
    }
复制代码

4. 处理需要发送的消息

在步骤3，我们处理的是上行的消息，会涉及到解码、业务处理、编码、应答等步骤。接着我们需要处理发送的消息，即下行的消息。

Downlink message processing will be relatively simple, as long as you get the corresponding MQTTClientinstance and protocol processor, you can encode it, and then send the message

@Slf4j
public class MQTTPublish {
    private MQTTClient client;
    private MQTTProtocolHandler protocolHandler;
    public MQTTPublish(MQTTClient client, MQTTProtocolHandler protocolHandler) {
        this.client = client;
        this.protocolHandler = protocolHandler;
    }
    public void publish(MQTTProtocol protocol, byte[] data){
        byte[] content = protocolHandler.encode(protocol, data);
        String deviceId = protocol.getDeviceId();
        String topic = protocol.getCmd().getTopic().getTopic();
        client.publish(topic, deviceId, content);
    }
}
复制代码

The above code can only handle the asynchronous downlink protocol. In some scenarios, the downlink protocol also needs to wait for the response from the device. At this time, this code cannot meet the needs.

Therefore, we also need to repackage this code. We design an extension point, different business types have different sending logic

public interface MQTTPublishHandler extends IExtensionHandler<BusinessType> {
    <T extends BaseCmd, C extends BaseCmd> T handle(C cmd, Class<T> clazz);
}
复制代码

Then deal with its implementation class.

@Override
    public <T extends BaseCmd, C extends BaseCmd> T handle(C cmd, Class<T> clazz) {
        CmdEnum cmdEnum = CmdEnum.get(cmd.getCmd());
        // 编码
        EncodeCmdHandler<C, T> handler = factory.getExtensionHandler(cmdEnum, EncodeCmdHandler.class);
        byte[] data = handler.encode(cmd);

        // 根据业务类型，拿到具体的协议处理器
        MQTTProtocolHandler protocolHandler = factory.getExtensionHandler(BusinessType.CHARGING, MQTTProtocolHandler.class);
        MQTTPublish publish = new MQTTPublish(client, protocolHandler);
        Long serial = this.getSerial(cmd.getDeviceId());

        // TODO 这里是具体的实现类，需要具体业务实现
        ChargingMQTTProtocol protocol = new ChargingMQTTProtocol();
        protocol.setSerial(serial.shortValue());
        protocol.setDeviceId(cmd.getDeviceId());
        protocol.setVersion("10");
        protocol.setMac(cmd.getDeviceId());
        protocol.setCode(cmd.getCmd());
        protocol.setCmd(cmd);
        publish.publish(protocol, data);

        // 阻塞应答
        RedisMessageTask task = new RedisMessageTask();
        RedisMessageListener listener = new RedisMessageListener(task);
        try {
            // 配置RedisKey
            String key = MQTTRedisKeyUtil.callbackKey(cmd.getTopic().getTopic(), cmd.getDeviceId(), serial);
            ChannelTopic topic = new ChannelTopic(key);
            redisMessageListenerContainer.addMessageListener(listener, topic);
            // 同步阻塞
            Message message = (Message)task.getFuture().get(60000, TimeUnit.MILLISECONDS);
            return JsonUtil.fromJson(message.toString(), clazz);
        }catch (Exception ex){
            log.error("消息获取失败: {}", ex.getMessage(), ex);
            throw new CommonBusinessException("-1", "Redis应答失败: " + ex.getMessage());
        } finally {
            redisMessageListenerContainer.removeMessageListener(listener);
        }
    }
复制代码

Here is the use of Java CompletableFutureclasses for asynchronous blocking. In addition, we implement a monitor in Redis by using the MQ mechanism of Redis. When there is an upstream message as a downstream response, we will StringRedisTemplate#convertAndSendsend a message, monitor the received message, and set it CompletableFutureto respond to it.

RedisMessageTaskWill hold a CompletableFuture instance and a reference to RedisMessageListener. Its code is as follows:

@Data
public class RedisMessageTask<T>{
    private CompletableFuture<T> future = new CompletableFuture<>();
    // Redis的监听器
    private RedisMessageListener listener;

}
复制代码

RedisMessageListenerHold RedisMessageTaskthe reference, when the message is received, set the message to CompletableFuture, and CompletableFuturethe blocking of the instance will receive a reply.

public class RedisMessageListener implements MessageListener {
    private RedisMessageTask task;
    public RedisMessageListener(RedisMessageTask task) {
        this.task = task;
    }
    @Override
    public void onMessage(Message message, byte[] bytes) { ;
        task.getFuture().complete(message);
    }
}
复制代码

Finally, the message can be sent when the uplink response is received.

stringRedisTemplate.convertAndSend(key, JsonUtil.toJson(data));
复制代码

finally

Well, the above are some notes about the front-end time contacting MQTT related content. The related code is still partially coupled with the business because more detailed design and decoupling have not been done during the implementation process. However, it will also be organized into a Lib package and placed in the Anyin Cloud project in the future, so stay tuned.