Intelligent environmental protection systems usually involve a large number of sensors and monitoring equipment to collect, analyze and process environmental data. These data are usually time series data , that is, data generated in chronological order over a period of time. They are large-scale and require fast and accurate processing. Analyze and process. Therefore, time series data processing is an important problem faced by intelligent environmental protection systems. Many projects adopted traditional big data solutions at the beginning of their creation. As the volume of data grows, problems such as poor performance, low efficiency, and high costs gradually emerge. . In this article, we summarize the data architecture upgrade practices of three typical smart environmental protection projects to provide reference for enterprises in need.
Zhongke Huiruan x TDengine
"In the past, we used traditional database architecture design for smart environmental protection projects. Real-time statistics and analysis of massive second-level monitoring data took a long time, low CPU and memory utilization, and disk IO overload. In the smart environmental protection Internet of Things application project in City A, we innovatively used With TDengine , real-time streaming computing related functions are used to solve big data storage and calculation, reduce the complexity of code development, make operation and maintenance work extremely simple, and greatly reduce operating costs."
business background
The "Smart Environmental Protection" Internet of Things application project in City A undertaken by Zhongke Huiruan requires the collection of monitoring data generated by various sensing devices and the various operating states of the monitoring equipment. The project collects more than 200 million pieces of various monitoring data every day. If the company's original The architecture can barely store daily data, but if you need to implement multi-latitude grouping aggregation queries with conditions similar to "how many muck trucks passed Section A at two o'clock in the afternoon on a certain day", then it is impossible to use traditional databases to achieve this type of query Check the requirements. Considering that various types of sensing monitoring equipment will generate a large amount of second-level and minute-level monitoring data storage and real-time calculation at all times, after comparing the performance and stability of multiple time series databases ( Time Series Database ), in the end, Zhongke Huisoft TDengine is used to store and calculate ecological environment monitoring data in real time.
Zhongke Huiruan selection test results
Click on the case to view more technical details
Kunyue Internet x TDengine
"After applying TDengine, when performing real-time calculation of the average volume per minute, we only need to simply define the time window and sliding increment, and the database can return the average volume per minute; in terms of processing real-time monitoring and early warning indicators, Stream calculations can be established specifically for this type of data, and the calculation results can be written to a new table (strm_pt_0304 below) for storage, so that the results of the entire real-time calculation can also be reviewed historically. Many numerical calculations that originally needed to be processed in the program are now It is entirely borne by TDengine, which not only shares the computing pressure of the program, but more importantly, the aggregation results can be automatically and persistently stored, supporting instant review of historical data."
business background
Kunyue Internet's "a Environmental Protection" APP is based on the self-built environmental protection industry Internet platform (INECO platform), which processes and analyzes massive data on environmental infrastructure in real time. It can collect data on various monitoring indicators of industrial atmospheric environmental protection in real time in seconds, respectively. The massive data collected can be analyzed and displayed in three dimensions: year, month, and day, combined with different collection frequency cycles. During the database selection, after comparing Alibaba's time series database TSDB , traditional MySQL and TDengine, TDengine finally stood out with its efficient performance and unique design ideas.
Architecture diagram
Click on the case to view more technical details
Guangdong Institute of Environmental Sciences x TDengine
"We store 7.65 billion pieces of data in one super table (a total of 16 billion pieces of data in four super tables), scattered in 19,419 sub-tables, with an average of 390,000 rows in each table. Due to the characteristics of TDengine super table, plus columns With special storage and ultra-high compression capabilities, these data only occupy 240GB of memory, which not only helps us save a lot of storage space, but also lays a good foundation for data query performance: for a super table with 7.6 billion rows, the grouped TOP query only It took 0.2 seconds; based on TDengine, it returned 2,968 rows and only took 0.06 seconds."
business background
In order to solve the management of ecological and environmental data in domestic environmental quality management, pollution source supervision and digital government, the Guangdong Academy of Sciences has created an ecological and environmental data governance service project to help enterprises connect all relevant business information systems and establish data warehouses. Different from general data storage requirements, the storage solution of the perception layer of this project has higher requirements for data reading and writing frequency and low latency. At the same time, due to the huge amount of data, it also requires higher storage efficiency. Previously, a relational database was used for this purpose. Data storage can only retain data for 3-5 days at most, and old data has to be deleted on a daily basis. Later, I considered using the TimescaleDB extension of PostgreSQL, but it did not meet the independent and controllable requirements of government informatization. After a long period of research and testing, TDengine was successfully implemented.
Architecture diagram
Click on the case to view more technical details
Conclusion
Judging from the above-mentioned corporate practices, the application of TDengine in intelligent environmental protection projects has great advantages. It can realize efficient data collection, storage, analysis and display functions, and provide strong technical support for environmental protection systems. At present, the fully managed time series data cloud service platform TDengine Cloud has also been launched. The ultimate elastic scaling can further improve the cost reduction and efficiency increase of the business. Everyone is very welcome to try it~ If you are facing data processing problems, you can also add Xiao T WeChat (tdengine), apply to join the TDengine technical exchange group, and discuss solution paths with like-minded developers.
To learn more about the specific details of TDengine Database , you can view the relevant source code on GitHub .