The practice of YoMo combined with CluingOS in Industrial Internet

foreword

YoMo is an open source programming framework, built for low-latency streaming data processing in the field of edge computing. Its bottom layer is based on HTTP 3.0's core communication layer IETF QUIC protocol communication, and uses Functional Reactive Programming as a programming paradigm to facilitate developers to build reliable, Real-time computing application of secure time series data, optimized for 5G and WiFi-6 scenarios, releasing real-time computing value.

Y3 is a Golang implementation of YoMo Codec that describes a fast and low CPU-cost codec focused on edge computing and stream processing. Check out the explainer for more info on more ways to combine with YoMo .

CluingOS is a cloud-native hyper-converged industrial IoT platform with Kubernetes as the core. Its architecture can easily integrate, integrate and install third-party applications with cloud-native ecological components, and supports cloud-native applications in multi-cloud and multi-cluster applications. unified distribution and operation and maintenance management. .

In this case, we combine the low-latency streaming processing of YoMo +Y3 with the distributed deployment features of CluingOS, and show how to develop and deploy an efficient industrial data collection application system to experience the collection of sensor data from the edge. The entire process of transferring data to the cloud for data streaming over more than 2,000 kilometers in a delayed and efficient manner. Based on this case, you can develop application scenarios that meet your own needs.

predicate

• xxx-source: Indicates a data source collection program that can directly receive data of the MQTT protocol.
• xxx-zipper: Represents a workflow and control plane.
• xxx-flow: Represents a workflow unit, used for actual business logic processing, scheduled by zipper.
• xxx-sink: Indicates the destination of a data transmission. In this case, it is a WebSocket service that consumes data and is scheduled by zipper.
• xxx-web: Represents a web service that displays real-time sensor data.

Architecture

As can be seen from the case in the figure, two independent service areas of the edge and the cloud are distinguished. The edge is located in Shanghai and the cloud is located in Guangzhou, with a distance of more than 2,000 kilometers. We can see its low-latency in the final test. Streaming offers surprising optimizations for the collection and processing of data. In addition, the containerized distributed deployment of the CluingOS hyper-converged industrial IoT platform can efficiently deploy and debug our applications, really fragrant! The following is a brief introduction to each module and service:

sensor

Vibration sensor, membrane key sensor

• Shock sensor. It is used to generate vibration-related raw data and is converted to MQTT protocol data through the Lora receiver. The data format is as follows:

  • TOPIC：shake/20210627_cluing/S07

  • Payload：

    {
    	// 租户数据库实例
    	"tenantId": "20210627_cluing",
    	// 采集设备终端DEVEUI
    	"devEui": "0850533277387820",
    	// 采集原始数据
    	"data": "CwcMys+69Iks0As4YS4N6A==",
    	// 采集数据时间
    	"createDate": 1624937248919,
    	// 采集温度
    	"temperature": 75,
    	// Z轴振动强度
    	"vertical": 81,
    	//X轴振动强度
    	"transverse": 53
    }
    

• Membrane key sensor. It is used to generate the raw data related to the key, and it is converted to the data of the MQTT protocol through the Loar receiver. The data format is as follows:

  • TOPIC：shake/20210627_cluing/S05

  • Payload:

    {
      	// 租户数据库实例
    	"tenantId": "20210627_cluing",
      	// 采集设备终端DEVEUI
    	"devEui": "393235307d377504",
      	// 采集原始数据
    	"data": "AAAQ5gAAAQIIAA==",
      	// 采集数据时间
    	"createDate": 1624937248919,
      	// 按键设备按键值
    	"key": "0800"
    }
    

5G CPE All-in-One

Lora receiver, forwarding engine, shake-source, shake-web

5G CPE 一体机It is a gateway device deployed on the edge. It can receive sensor data and convert it into MQTT protocol data. Our YoMo edge receiver shake-source is deployed on this gateway device. Its major feature is that it can accept CluingOS The container deployment and resource scheduling of the hyper-converged industrial IoT platform makes it easy to distribute applications to this gateway device even if you are thousands of miles apart, without the need for remote login to operate.

• The Lora receiver is the default service for the gateway. It can receive monitoring data from various sensors and convert it into MQTT protocol. The data of different devices can be divided into different themes and easy to manage. The data of the vibration sensor and membrane button sensor in the previous section are converted to the definition shown data format.
• The forwarding engine is the default service for the gateway. MQTT data can be forwarded to different MQTT Broker services, of course, including shake-source in our case.
• shake-source data source. The data source receiving service developed based on the YoMo framework, its function is to convert the data of the MQTT protocol into the Y3 data format and transmit it to the shake-zipper workflow engine in the cloud in the way of QUIC .
• shake-web data display. This is a display web service that displays the real-time data of two sensors, mainly consuming the WebSocket data provided by shake-sink, and can also display a complete real-time RTT round-trip delay.

SaaS service

shake-zipper, shake-flow, shake-sink, HOMEY Humei event lean management system

This is a complete set of cloud services. The containerized deployment is also scheduled and managed by CluingOS. You only need to log in to CluingOS with different users to switch and manage services on the edge or in the cloud.

• shake-zipper workflow engine. Through orchestration (workflow.yaml), multiple flows and sinks can be scheduled, allowing them to connect business logic in a flow manner to meet complex requirements. All communications and codecs connected to it are carried out with QUIC+Y3, providing reliable real-time streaming processing, and experiencing the fun of streaming programming throughout.
• shake-flow logic processing unit. In this case, the processing unit decodes the data transmitted from the source into Topic and Playload, respectively processes the data of the vibration sensor and the membrane button sensor, and calls the HOMEY management system to give an alarm after reaching a certain threshold. Or issue a control command to control the edge device.
• shake-sink data output unit. In this case, it does not output to the database, but builds a WebSocket server to output real-time sensor data to any web page for display and consumption. The data here is displayed as the data source of shake-web.
• HOMEY Humei management system. After receiving the shake-flow event notification, the Humei system will send out alarm information or issue control commands to the edge to control some devices to perform certain operations. But in this case, we will use the time when the Humei system receives the event notification as the statistical end point of the delay, and analyze that our low delay based on YoMo+Y3 has indeed been greatly optimized.

CluingOS Hyper-converged Industrial IoT Platform

Logging in as a different user can switch the deployment and management of edge services or cloud services

CluingOS provides industrial IoT big data intelligent platform services and an end-to-end integrated transparent factory system of "on-site collaboration + process control + data intelligence" with containerized, subscription-based, and micro-service architecture, supporting private cloud, public cloud or hybrid cloud A variety of ways, distributed deployment implementation.

code

The project yomo-cluing-shake provides a complete set of source code. The following table provides a brief description of each module for interested friends to view. With reference to the code of this case, it is easy to develop a case similar to the scene.

module	address	run locally	illustrate
zipper	zipper	make debug_zipper	Orchestrate the workflow and data flow of this case
flow	flow	make debug_flow	Preprocessing and alerting on sensor data
sink	sink	make debug_sink	Provide WebSocket service for data display
source	source	make debug_source	Collect sensor data in MQTT message format
emitter	emitter	make debug_emitter	Simulation generates vibration and keystroke data
web	web	make debug_web	Consume a WebSocket service to display sensor data
anything-mqtt	yomo-source-mqtt-starter		Develop common components for xxx-source

Containerization department

By downloading the project code in the previous section, you can quickly run it locally and experience the fun of YoMo development. At the same time, we provide the Dockerfile files corresponding to each module to package the corresponding images and upload them to hub.dockder.com for CluingOS to deploy.

module	Dockerfile	mirror address	The latest version
zipper	Dockerfile.zipper	shake-zipper	yomorun/shake-zipper:latest
flow	Dockerfile.flow	shake-flow	yomorun/shake-flow:latest
sink	Dockerfile.sink	shake-sink	yomorun/shake-sink:latest
source	Dockerfile.source	noise-source	yomorun/shake-source:latest
emitter	Dockerfile.emitter	shake-emitter	yomorun/shake-emitter:latest
web	Dockerfile.web	shake-web	yomorun/shake-web:latest
anything-mqtt		yomorun/quic-mqtt	yomorun/quic-mqtt:latest

yomorun/quic-mqtt:latest is the basic image for packaging xxx-source, which can quickly package custom code, but it can be ignored for the time being in this case.

CluingOS Deployment

Containers on the edge and the cloudCluingOS工业超融合系统 can be managed separately by logging in with different users .

Deploy cloud services

User A deploys the zipper/flow/sink service to the cloud machine.

1. Create custom apps.

   Create a homebrew application shake-cloud to manage services in the cloud.

2. Create a service.

   Enter the console of the shake-cloud application, select Add Service, select Stateless Service, and then enter the process of creating a service.

   

   • Basic information: The name of the specified service is shake-sink

   • Container image:

     • Add container image: choose to search shake-sink from DockerHub

     

     • Port settings: Specify the service port exposed by the container, for example: 8000
     • Environment variables: For example SHAKE_ZIPPER_ADDR=shake-zipper.yomo-cluing-shake:9000, here shake-zipper.yomo-cluing-shakeis the internal DNS name of the zipper obtained after creating shake-zipper.

   • Mounted Storage and Advanced Settings: Neither is required in this case.

   • Edit external network access: select the access method of NodePort, and obtain the externally exposed port number 30095

3. service list

   So far, the services of zipper/flow/sink have been created respectively.

   

Deploy edge services

User B deploys the source/web service to the edge 5GCPE all-in-one machine.

The deployment method is the 部署云端服务same, first create a shake-edge application, and then create the corresponding stateless service in the application to obtain the service list:

Access sensor display effect

The forwarding engine forwards the MQTT message to the shake-souce service.

In order to access sensor data, it is only necessary to modify the forwarding address of the forwarding engine to the address port of the shake-source service, and the received real-time data can be displayed through shake-web .

Effect comparison

In order to compare the delay with the traditional http report data, a test case is designed to compare the effect.

This integration with yomo is carried out in a real environment. All relevant application components are deployed and installed in containers using CluingOS. The scenarios cover cloud-edge-end applications. In addition to verifying the acceleration effect of yomo, the integration of yomo has been integrated into the Lingxi platform system (CluingOS/AIOT/MOM). The following mainly introduces the acceleration test effect of yomo.

actual scene

As can be seen from the test architecture diagram, two sets of tests are used for comparison:

QUIC+Y3 channel

As shown in the orange process in the figure, the data transmission path is:

HTTP channel

As shown in the green process in the figure, the data transmission path is:

Environmental description

• The 5G CPE all-in-one machine is placed in the Shanghai Jinqiao Office.
• The MOM intelligent manufacturing operation management system is deployed in Guangzhou Tencent Cloud.
• The test delay link is the network from Shanghai Jinqiao Office to Guangzhou Tencent Cloud .

testing method

• Prepare sensor A (3932353062376611) and sensor B (0850533277387820).
• Sensor A uses the traditional HTTP protocol, and Sensor B uses the QUIC+Y3 protocol.
• Each sensor transmits 600 records and calculates the average delay.

Calculation formula

select device_sn,(timeCount/count)*1000 agv,count from(
  select device_sn,sum(unix_timestamp(end_time) -unix_timestamp(begin_time)) as timeCount,
  count(id) as count from sdm_device_logs_copy2 GROUP BY device_sn
) t

Test conclusion

serial number	agreement type	equipment number	Average Latency (ms)	Number of test records
1	HTTP	Sensor A (3932353062376611)	100.268333(ms)	600
2	QUIC+Y3	Sensor B (0850533277387820)	33.493333(ms)	600

Using the calculation formula from the collected samples, the average delay of each sensor transmitting 600 pieces of data is obtained. It can be seen that the average delay of the HTTP protocol is 100.26 milliseconds , and the average delay of the QUIC+Y3 protocol is only 33.49 milliseconds , which has a very obvious acceleration. Effect.

concluding remarks

In recent years, the new generation of information technology has developed by leaps and bounds, and the Internet has accelerated from the consumer field to the industrial field. From the perspective of digital industrialization, the Industrial Internet needs new technologies, products and solutions to develop to a wider, deeper and higher level. The YoMo open source programming framework can greatly improve the transmission efficiency from the edge to the cloud, improve real-time performance and obtain the advantages of low latency, and at the same time, the new streaming computing and programming paradigm gives developers a new development experience, which is more natural and efficient. Develop applications for streaming computing. With the help of Lingxi 's CluingOS hyper-converged industrial IoT platform, container services can be quickly and easily deployed on the edge and the cloud, and service governance capabilities can be greatly improved.

{{o.name}}

{{m.name}}