Basic concepts of IoT network middleware

The Internet of Things network middleware is developed using Java language and based on open source projects such as netty, spring boot, redis, etc. It supports underlying protocols such as udp and tcp communication, as well as http, mqtt, websocket (default implementation and custom protocol header implementation), modbus( tcp, rtu), plc, dtu (supports heartbeat, device registration function, AT protocol and custom protocol support), dtu for modbus tcp, dtu for modbus rtu component adaptation and other upper-layer protocols. Focus on the underlying network interaction and equipment of the Industrial Internet of Things Management, data storage, big data processing. (The plc includes Siemens S7 series, Omron Fins, Rockwell CIP, Mitsubishi MC). Data storage will use taos database and redis message queue

Main features

Support the server to start monitoring multiple ports, and unify the API interfaces that can be used by all protocols
Contains a set of proxy client communication protocols, supporting calls: client->server->device->server->client
Supports the separation of device protocol objects and their business objects (supports default business processor [spring singleton injection] and custom business processor)
Supports synchronous and asynchronous calls to the device, supports synchronous and asynchronous calls between the application proxy client and the device server, and the three ends of the device
The server supports device online/offline/abnormal event notifications, supports custom heartbeat events, and the client supports disconnection and reconnection.
Rich log printing functions, including device online and offline prompts, the life cycle of a protocol (request or request + response), etc.
If the connection is disconnected during support request, it will automatically reconnect (sent after synchronization is successful)
Supports the automatic creation of a client if the client does not exist when sending a request (sent after the synchronization is successful)
Supports serving as an mqtt gateway, publishing data collected from the Industrial Internet of Things to the mqtt server more easily and conveniently
Supports common IoT protocols such as: mqtt, plc, modbus, websocket
Supports operating plc using modbus protocol through dtu mode

Simulation tools

QtSwissArmyKnife supports debugging of udp, tcp, modbus, websocket, serial port, etc.
IotClient supports simulation and debugging of plc (Siemens, Omron, Mitsubishi), modbus, serial port, mqtt, tcp, udp, etc.

Development documentation

1. Definition of terms

Message object (Message): Message is an encapsulation of binary data transmitted in the network and is also the carrier of binary data. To a certain extent, message = binary data
Protocol object (Protocol): The protocol is a normative constraint of the message. For example, the message content is: 0x04 AF CD EE 03. How do you know what this string represents? The protocol is a statement of this string of data, such as The first byte represents the length of several bytes after the data, the second byte is the voltage, the third byte is the current, and the fourth byte is the check bit.
Component object (FrameworkComponent): On the server side, the component is used to manage the various interfaces required by a port; on the client side, the component is used to manage all interfaces connected to the same server and all connected clients. For example, the device manager on the server side, the codecs required for messages, and synchronous and asynchronous processing are all managed by components.
Synchronization: When calling a request, the request thread will be locked and blocked until a response is received or a timeout occurs to unlock it.
Asynchronous: When calling a request, the requesting thread returns directly after sending the message. It does not block the calling thread. Instead, it registers a callback function and waits until the other party responds or times out before doing business processing.
Codec: a processing object used to unpack and glue binary data on the network
Protocol Factory: a place used to create each protocol object (because a client may contain multiple functions (protocols), and each function corresponds to a protocol object, then there will be many protocol objects. The protocol factory is used to manage protocol objects. creation)
Protocol handler (ProtocolHandle): used for business processing of protocols

The following is an example to illustrate the use of the IoT framework : For example, the server receives the client message as follows: 0x01 11 12 13 05 06 EE FF eight bytes. If the client does not send continuously after sending this message, The data received by the server is a complete packet. We can easily read the contents of the buffer and process it; if the server does not read the buffer in time after the first packet is sent, it is still busy with other things. Area content At this time, the second message from the client was received. At this time, the data in the data buffer is as follows: 0x01 11 12 13 05 06 EE FF 02 21 22 23 25 26 AA BB CC DD. At this time, the buffer read by the program The area contains two complete packets of packets. How does the program separate and process the two packets of packets at this time? This is when the codec comes into play! How to deal with the above situation of packet sticking and unpacking? netty provides several commonly used decoders

2. Codec

FixedLengthFrameDecoder: Fixed length decoder is the simplest way. The length of each packet is fixed. For example, each packet is 8 bytes, then the program can unpack in units of 8 bytes.
LineBasedFrameDecoder: The newline decoder allows each message to end with a newline character. At this time, the program can loop through each byte to determine. If this byte is a newline character, it means that a packet of messages has been read.
DelimiterBasedFrameDecoder: If our data happens to contain a newline character, an error will occur when reading. At this time, a custom delimiter can be used to split the message.
LengthFieldBasedFrameDecoder: Whether it is newline decoding or custom delimiter decoding, each byte needs to be judged in a loop. If a complete packet is very long, the performance will be very poor. At this time, there is a very easy-to-use and high-performance Extremely high decoder, length field decoding, is to add a length field to the message to identify the length of the entire message.
ByteToMessageDecoder: If there is a better decoding method, you can use custom message decoding
SimpleChannelInboundHandler: simple decoder

When connecting devices, manufacturers generally provide protocol documents, and then we need to choose the appropriate decoder. After we confirm the decoder, we can start encoding. Let’s start with the server-side docking tutorial.

3. Server-side tutorial

When writing a server-side network program, we need to listen to a certain port to connect the client. When we select a decoder, we can select the corresponding server-side decoder component to open a certain port. The IoT framework adapts to several features provided by Netty. commonly used decoders

Create decoder component

FixedLengthFrameDecoderServerComponent A server-side component that uses a fixed-length decoder
LineBasedFrameDecoderServerComponent A server-side component that uses a newline decoder
DelimiterBasedFrameDecoderServerComponent A server-side component that uses a custom delimiter decoder
LengthFieldBasedFrameDecoderServerComponent A server-side component that uses the length field decoder
ByteToMessageDecoderServerComponent A server-side component that uses a custom decoder
DatagramPacketDecoderServerComponent server component of udp protocol
SimpleChannelDecoderComponent The component corresponding to the simple custom decoder

The following is an example using LengthFieldBasedFrameDecoderServerComponent

// 首先：必须先创建一个组件对象来继承LengthFieldBasedFrameDecoderServerComponent // 以iot-test模块的断路器服务端模拟为例 public class BreakerServerComponent extends LengthFieldBasedFrameDecoderServerComponent<BreakerServerMessage> { public BreakerServerComponent(ConnectProperties connectProperties) { super(connectProperties, ByteOrder.LITTLE_ENDIAN, 256, 0 , 4, 0, 0, true); } xxx 实现省略 } // 注：要求传入ConnectProperties对象作为构造参数，此对象可以指定ip和端口

We see that the above component requires a generic parameter BreakerServerMessage. This parameter is the message object. As we said above, the message object is a binary data carrier for use in various objects of the IoT framework. Let's take a look at the report. In addition to being a data carrier, what other extended functions does the text object have?

Create message object

// 创建服务端报文对象必须继承ServerMessage类 public class BreakerServerMessage extends ServerMessage { public BreakerServerMessage(byte[] message) { super(message); } //省略其他构造函数 @Override protected MessageHead doBuild(byte[] message) { this.messageBody = MessageCreator.buildBreakerBody(message); return MessageCreator.buildBreakerHeader(message); } }

First, the constructor of the message object BreakerServerMessage#BreakerServerMessage(byte[]) must exist
Message objects are the bridge connecting components and protocols, so a corresponding message object needs to be created for each server component.
It is necessary to initially parse out the corresponding equipmentCode, messageId, and protocolType parameters in the BreakerServerMessage#doBuild(byte[]) method. This is also a preliminary parsing of the client request data.

After we create the component and message class, we can start the application. At this time, the log will print out that the port configured by the component has been opened for listening. We have made a good start here and it is half successful. The next step is to create the protocol object. The protocol document obtained from the manufacturer contains at least one protocol. Generally, we recommend creating a corresponding one for each protocol in the document. Protocol object (Protocol)

Create protocol object

The protocol object is used to parse the received binary data into fields corresponding to the protocol document; due to the needs of the framework architecture, we divide the protocol into two types as follows:

ClientInitiativeProtocol declares that this protocol is a protocol actively initiated by the client, such as a circuit breaker actively reporting the current current, voltage, or alarm.
ServerInitiativeProtocol declares that this protocol is a protocol actively initiated by the server. For example, the server issues an instruction to open the circuit breaker.

First, let’s take a look at the ClientInitiativeProtocol method declaration, again taking the circuit breaker as an example.

// 用来接收断路器主动上报的电压电流等数据 public class DataAcceptProtocol extends ClientInitiativeProtocol<BreakerServerMessage> { private double v; // 电压 private double i; // 电流 private double power1; // 有功功率 private double power2; // 无功功率 private double py; // 功率因素 public DataAcceptProtocol(BreakerServerMessage requestMessage) { super(requestMessage); } @Override protected void doBuildRequestMessage(BreakerServerMessage requestMessage) { byte[] message = requestMessage.getBody().getMessage(); this.v = ByteUtil.bytesToInt(message, 0) / 100.0; this.i = ByteUtil.bytesToInt(message, 4) / 100.0; this.power1 = ByteUtil.bytesToInt(message, 8) / 100.0; this.power2 = ByteUtil.bytesToInt(message, 12) / 100.0; this.py = ByteUtil.bytesToShort(message, 16) / 100.0; } // 响应断路器的请求 @Override protected BreakerServerMessage doBuildResponseMessage() { Message.MessageHead head = requestMessage().getHead(); return new BreakerServerMessage(MessageCreator.buildBreakerHeader(head .getEquipCode(), head.getMessageId(), 4, head.getType()), MessageCreator.buildBreakerBody(StatusCode.Success)); } // 省略其他 }

The platform can already receive data actively reported by the device, so how can the platform actively send data to the device? The ServerInitiativeProtocol protocol is used to declare that a protocol is actively sent to the client by the platform. The following is an example of a protocol sent by the platform to a circuit breaker switch.

/** * 切换断路器的开闭状态 */ public class SwitchStatusProtocol extends ServerInitiativeProtocol<BreakerServerMessage> { private String deviceSn; public SwitchStatusProtocol(String deviceSn) { this.deviceSn = deviceSn; } /** * 构建要发送给断路器的报文 */ @Override protected BreakerServerMessage doBuildRequestMessage() throws IOException { DefaultMessageHead messageHead = MessageCreator.buildBreakerHeader(Long.valueOf(this.deviceSn), 0, protocolType()); return new BreakerServerMessage(messageHead); } /** * 处理断路器对此处请求的响应 */ @Override protected void doBuildResponseMessage(BreakerServerMessage message) { /*设备响应是否切换成功的处理*/ } @Override public BreakerProtocolType protocolType() { return BreakerProtocolType.SwitchStatus; } } // 然后在业务代码里面调用请求方法：new SwitchStatusProtocol(deviceSn).request(); 这样就可以向指定的设备发起请求了