In our "Overseas Energy Storage Project" project, we need to monitor battery safety in real time, collect and record the charge and discharge process, current/voltage and other values of each use. Such data are timestamped and are typical time series data . . In order to cope with the massive amount of user usage data in the future, we need to choose a professional time series database ( Time Series Database ).
Our business is overseas. Last year, version 2.x was deployed overseas locally. However, since the maintenance team is located in China, there was a lot of trouble in network communication first. Secondly, since we are deploying the open source version of TDengine , we need to deploy and optimize it ourselves, study documents, feedback issues through the community, etc., which costs a lot of time. Later, TDengine Cloud , the time series data cloud platform officially released by TDengine earlier this year, came into our view. Since our current business volume is not large, the most intuitive help for us from TDengine Cloud is: full hosting.
The cloud service comes with the same level of services as TDengine Enterprise Edition, so we no longer need to worry about deployment, optimization, expansion, backup, off-site disaster recovery and other matters. This reduces the burden on our developers and can focus entirely on the core business. Since our equipment volume is temporarily small, according to the official existing pricing rules, the basic version will suffice. After calculation with the billing plan estimator, we finally chose the basic version specification of 1,200 yuan/month.
We build a separate table for each energy storage device. For one type of energy storage device, we build a super table, including power consumption, charging capacity, power usage status, charging status and other indicators. There are more than 100 columns in total, and each device is available 7*24h. A row is written to the database every 10 seconds. Through the "Data Browser" page, you can easily see the metadata information of the library/table:
Writing capability analysis
TDengine's powerful innovations of one device and one table, columnar compression, and tags, specially designed based on the characteristics of time series data, fundamentally solve the problem of locking required for data writing, low efficiency of row compression, and redundant static data. Storage problems.
Our data processing process is shown in the figure below. The time series data generated by certain types of energy storage equipment will be uploaded in MQTT mode. The business data is forwarded to PostgreSQL, and the time series data generated by the equipment, equipment operation logs, and equipment status data are transferred to TDengine. Each system in the middle office will uniformly plan to use the data in these databases for analysis and calculation, and can also directly control the equipment to issue instructions. Finally, it is reflected on the front end with the help of software methods such as PC Web, APP and other management platforms.
During the testing phase, TDengine's data compression rate can easily reach within 10%, and millions of rows of data can be written per second. This writing effect is also well achieved in specific practice.
In addition to writing and compression performance, TDengine's query capabilities are also of great concern to us.
Query effect analysis
In order to provide high-quality after-sales service and improve user experience, and use the above written data scientifically and rationally, we will make many types of queries, such as monitoring the health status of electrical products, analyzing equipment power consumption trends, service life, etc. wait.
Here are a few typical queries:
1. Use the session window to count the sum of the input power of a single device in each continuous time interval less than or equal to 1 minute:
select FIRST(ts) firstTs, LAST(ts) lastTs, count(*) countVal, sum(input_total_power) totalPower
from device_data_58CF7920B63C where ts >= '2023-08-09T00:00:00.000Z'
and ts< '2023-08-10T00:00:00.000Z' SESSION(ts, 1m)
2. View the trend of device input power through interval, and use the offset time zone offset:
select _WSTART as ts, avg(inverter_power) avgVal, min(inverter_power) minVal, max(inverter_power) maxVal
from gendome.device_data_58CF7920B63C where inverter_power > 0 and ts >= '2023-08-09T00:00:00.000Z'
and ts< '2023-08-10T00:00:00.000Z'
interval (1d, 960m) FILL (NULL)
Since our application and TDengine Cloud belong to the same AWS region, by using the Private Link function, our application network communicates privately with the cloud service without transmitting data through the public network, which greatly reduces write latency. At the same time, it also further saves the costs incurred by network traffic.
write at the end
TDengine has always been positioned as a "big data" processing platform. Especially version 3.0 has many underlying changes compared with 2.0, and can even support 1 billion tables.
For our relatively small scenario, TDengine Cloud’s pay-as-you-go and fully managed enterprise-level services allowed us to run this project at a very small cost, greatly increasing the product’s efficiency and retaining the Flexibility to expand at any time. In addition, interesting features such as data sharing and streaming computing are also waiting for us to dig deeper.
About Guoxuan Hi-Tech
Guoxuan Hi-Tech (stock code: 002074) is the world's leading new energy solutions provider, focusing on the research and development, production and sales of lithium-ion power batteries and energy storage batteries. Its comprehensive strength ranks among the top in China and even the world.
About the author of this article
Liu Zhicheng, platform architect of an energy storage product department of Guoxuan Hi-Tech, has been engaged in the energy Internet of Things industry for many years.