Build your own IoT platform from scratch
Features an IoT platform should have
- Data collection: Collect various data through sensors, equipment, etc., convert them into digital signals, and send them to the cloud for centralized management and processing.
- Data Storage: Store the collected data for subsequent analysis and processing. Storage methods generally include relational databases, non-relational databases, data warehouses, etc., and different storage methods are selected according to specific scenarios.
- Data cleaning and preprocessing: process the collected data, remove invalid data, abnormal data, etc., and perform preprocessing operations such as data normalization and data conversion.
- Data analysis and mining: Based on the analysis and mining of collected data, valuable information is extracted, which has an important impact on intelligent decision-making and forecasting.
- Data visualization and display: Visualize and display the analyzed and mined data, so that users can quickly understand the relevant situation of the data, so as to make better decisions and management.
design for these features
data collection
IoT platform projects are often faced with protocols and diversified messages. In view of this feature, we hope that this platform can access common protocols on the market such as MQTT, tcp, udp, CoAP, http, etc. For the message body, we should support a free parsing method. Here I plan to use a custom parsing jar method. For each new device, a different parsing jar is developed and uploaded to the server. After the message arrives, we call the corresponding jar to Parse the message.
data storage
For IoT projects, general data is divided into two types, business data and equipment reporting data, business data is stored in relational databases, and equipment data is stored in time-series databases
Data cleaning and preprocessing
Data cleaning and preprocessing are also placed in the parsing jar to filter some abnormal data, etc.
Data Analysis and Mining
At this point, consider connecting the data first, and then consider the design later.
Data Visualization and Presentation
It is tentatively displayed in the form of a large data screen
The entire platform technology architecture
Using the spring loud family bucket (using the micro-service version of Royi, Royi is very good, this part will not repeat the wheel), the call between services abandons the method of calling feign, and according to the characteristics of the Internet of Things project, prepare The publish-subscribe model is adopted to convert each business into different streams or time, the arrival and storage of device messages at the start and end of the stream, and other upper-level services subscribe to these different streams to achieve expansion. A way to achieve high cohesion and loose coupling.
some special points
Timing library insertion
Processing with relatively high pressure like this is intended to be implemented with responsive backpressure