Summary
This article mainly introduces the big data architecture of the shared energy storage SCADA system, how to use Honghu to better optimize the functions of the SCADA system, and how to empower users with self-service data analysis.
1. Introduction to shared energy storage
When it comes to shared energy storage, many of you may be unfamiliar with it. Let’s briefly introduce the value of shared energy storage and its future technological development direction.
1.1 The value of shared energy storage
Energy storage technology has great value. There are four basic functions of energy storage: one is peak shaving and valley filling, the other is virtual synchronization, the third is accurate controllability, and the fourth is response to grid commands and load control. The business model of energy storage was originally used by the power supply side, which was installed on the power supply side, but in June 2019, it was moved to the grid side and dispatched by the grid. It is called shared energy storage, and the shared energy storage is dispatched by the grid.
The "shared energy storage" business model fully takes into account the needs of all parties.
From an international point of view, Tesla will deploy a total of 4 GWh of energy storage projects in 2021, and its target of 1,500 GWh in 2030 has increased by 375 times in comparison. As market demand continues to exceed production capacity, the growth of Tesla's energy storage business is still limited by supply chain challenges.
In 2020, Tesla's US official website will officially launch the Megapack industrial-grade battery energy storage system. The price starts at $1 million, and it is only sold to enterprises, but it is still selling very well. In the earnings conference call for the second quarter of that year, Musk revealed that Megapack’s production capacity by the end of 2022 has been sold out.
Domestically, on April 21, 2021, the National Development and Reform Commission and the National Energy Administration issued the "Guiding Opinions on Accelerating the Development of New Energy Storage (Draft for Comment)" (hereinafter referred to as the "Draft for Comment"). The "Draft for Comments" proposes that by 2025, the transformation of new energy storage from the initial stage of commercialization to large-scale development will be realized, and the installed capacity of new energy storage will reach more than 30 million kilowatts.
By 2030, realize the comprehensive market development of new energy storage.
The figure below shows Bloomberg’s forecast on the installed capacity of various countries in the future shared energy storage market. We can see that whether it is China or the United States, the future energy storage market will be at least a trillion-level market, and the ceiling is high enough. In this era of surging new energy, shared energy storage has become a basic element of the new power system.
(Source: Bloomberg)
1.2 Battery energy storage is the future of technological development
Generally speaking, shared energy storage currently includes mechanical energy storage, electromagnetic energy storage, and battery (also called electrochemical) energy storage. From the table below, we can see that the energy storage technology based on lithium batteries has obvious advantages, and it is also the mainstream of shared energy storage at present and in the future.
Others include compressed air energy storage, high-energy capacitor energy storage, liquid flow battery energy storage, sodium-sulfur battery energy storage, lead-acid battery energy storage, etc. Due to various reasons such as geographical constraints and low specific energy, it is not an industry mainstream technology. Our shared energy storage SCADA system is also mainly based on lithium battery shared energy storage.
2. Traditional SCADA system
The so-called SCADA (Supervisor Control And Data Acquisition) data acquisition system is a comprehensive integrated control system integrating measurement technology, computer technology and communication technology, which can realize real-time central monitoring and management scheduling of shared energy storage systems.
A typical SCADA is shown in the figure below, which is divided into a station terminal and a management terminal. Station side: mainly consists of three parts: lower computer, communication network, and upper computer. Management terminal: generally includes pre-acquisition and SCADA applications.
Since the amount of data faced by early SCADA systems was relatively small, most of them used stand-alone or C/S architecture, and generally used relational databases, such as MySQL, SQL Server, etc. for storage. With the increase of collected data, such a system cannot save data for a long time, and will set a regular data cleaning strategy for several months. If long-term data comparative analysis is required, it will be difficult to meet the demand.
The following is a traditional SCADA system used by our customers:
3. Shared energy storage SCADA system based on big data platform
With the construction of various new energy (wind and solar) power plants, the data collection layer of the power plant has the characteristics of large data collection and high collection frequency. The history database will inevitably form complex and heterogeneous energy storage big data. Traditional SCADA The system began to face a series of problems such as computer CPU upgrade, insufficient memory, computer hardware expansion, and cost increase. Traditional data processing methods are difficult to quickly process massive shared energy storage big data.
Therefore, our shared energy storage customer proposes to build a shared energy storage SCADA system based on a big data platform.
3.1 Overall Architecture
Generally speaking, a shared energy storage power station is mainly composed of power transformation and distribution equipment, energy storage battery pack, energy management system (EMS), battery management system (BMS), energy conversion control system (PCS) and SCADA system.
During the construction, considering the complexity of the existing shared energy storage system and the large number of collection points, we adopted a time series database for the main data storage. In addition, the general requirement of a shared energy storage system is that it is in a closed environment and cannot connect to the Internet, so we will use a time server based on GPS (Beidou) for time synchronization.
The following figure is a schematic diagram of our shared energy storage SCADA system:
In order to more conveniently integrate some third-party standard collection boxes, we use the MQTT protocol as the main protocol for data transmission, so that the third-party standard collection boxes only need to be configured briefly to collect data.
In order to write acquisition programs more flexibly, we use the Python language for development. For data display and analysis, we separate the front and back ends. The back end is developed using the mainstream Java framework Spring Boot, and the front end is developed using Vue.
3.2 Data storage
The collected data is mainly stored in InfluxDB. For some special report requirements, we will periodically ETL data to the MySQL relational database.
For the storage structure of InfluxDB, the efficiency of some queries will be relatively low. Therefore, pre-computation will be performed in advance to meet the needs of fast data analysis. Of course, this is also a very common requirement for big data systems. Fortunately, InfluxDB provides the Rollup function, which can be conveniently pre-calculated according to the sampling accuracy and data aggregation method by writing SQL statements.
3.3 Live shooting
This is the first real shot of a shared energy storage power station that uses our SCADA system. The shared energy storage power station basically has the main functions of a common shared energy storage power station: load shaving, responding to grid commands and controlling loads.
3.4 System interface
The figure below is the main system diagram of our shared energy storage power station. All the data in the figure are collected and displayed in real time. In addition, according to the needs of customers, some simple system interaction control can be performed directly on the system diagram.
Generally, on the home page of the SCADA system of a shared energy storage power station, users hope to see the real-time situation of the energy storage system responding to the AGC command of the grid computer, so as to quickly analyze the current working power of the energy storage system. The picture below is the home page of a typical SCADA system we built:
4. Use Honghu to optimize the shared energy storage SCADA system
After learning that a new big data product has been launched in China, we hope to try it out. Of course, we have considered some other products before, but there is no significant improvement compared to our existing architecture in terms of price and ease of use.
4.1 Our Pain Points
Although our previous products based on the big data platform have made great progress compared with the original traditional SCADA system, there are still some unsatisfactory aspects.
4.1.1 Lack of flexibility
For the shared energy storage SCADA system, in addition to having the same monitoring function as the general SCADA system, it is more important that the shared energy storage SCADA system must respond to the order of the grid to accurately control the output of the energy storage system (that is, the energy storage system we mentioned earlier. The latter two of the four basic functions) provide data analysis services and assist decision-making.
About 3-6 months after the completion of the shared energy storage system, it is in the debugging period of the grid command response algorithm. In order to adjust and optimize the algorithm model, power engineers and algorithm engineers will continue to have a lot of temporary ideas. When the data changes after algorithm adjustment, users will further hope to have some new indicators or statistical charts presented.
Since all statistical analysis functions need to be implemented with Java code, once the user puts forward more data analysis requirements, it is necessary to re-schedule the requirements. If there is a WYSIWYG auxiliary data analysis tool, it can obviously better meet some temporary and exploratory analysis needs. Accelerate the debugging progress of the algorithm model of the shared energy storage system.
4.1.2 ETL dependencies
Although our core collection data is saved in InfluxDB, the current daily collection data volume of an energy storage power station is about 8-12GB. At present, after we build the first shared energy storage power station SCADA system and run it for one year, there are about 3TB of accumulated data. data. It is inefficient to directly query InfluxDB for some reports through SQL, so we will extract part of the statistical report data from InfluxDB to MySQL through ETL scripts to meet relatively complex analysis and report requirements.
4.1.3 Expansion is expensive
As the amount of data increases, if you want to use InfluxDB clusters in the future, the cost will be quite high. Since the community version of InfluxDB does not support the cluster mode, we have inquired about the InfluxDB Enterprise Edition that supports the cluster mode from relevant channels. It is said that the price can reach one million, which is obviously unaffordable for a SCADA system.
4.2 Selection comparison
Based on several factors such as improving flexibility, reducing reliance on ETL, and reasonable costs, we prefer ElasticSearch, Splunk and other products that can save the ETL process and model more flexibly. We just recently learned from friends in the industry A strong team in China has released a better product than Splunk - the Honghu Community Edition, and I can't help but want to try it out. Especially when we see the characteristics of Honghu's read-time modeling and vector acceleration, it can obviously solve our previous pain points.
4.2.1 Modeling at read time
According to the explanation of the official document, "mode when reading" is a technique for extracting useful fields (field) from raw data while searching for data. Since the ETL part can be omitted, this technology can greatly reduce the overhead of data import. This is very suitable for some temporary and exploratory needs of our SCADA system users.
Especially when the data collection enters the platform, the original data is directly stored, and by defining some column field extraction rules, the field values are dynamically extracted from the original data according to the referenced rules when querying.
(This picture comes from the official document of Honghu)
4.2.2 Vector Acceleration
Through the use, we also found a very important feature of the Honghu system, which is the use of vector accelerated computing. Vector computing can make full use of the SIMD (Single Instruction Multiple Data) instruction of the CPU. SIMD is a special type of CPU instruction. , this kind of instruction can operate multiple data at the same time in one instruction, which greatly improves the performance of data processing, which is why Honghu requires the CPU to support the AVX2 instruction set when installing.
4.3 New Architecture
Our new architecture will replace the relational database with Honghu, and it can even be used as the main data storage in the energy storage power station with a small installed capacity.
As shown in the figure below, through the integration with Honghu, the flexibility of the existing SCADA system has been improved, which is equivalent to directly empowering the original system with self-service analysis.
In the new architecture, we will let Honghu and InfluxDB work together. Using Honghu, on the one hand, replaces the role of MySQL and part of the precomputing processing of InfluxDB; on the other hand, we can allow users to directly log in to Honghu's management interface to perform some self-service data analysis, which is what users have been expecting before.
4.4 Project Integration
Having said so much, I can't wait to try the system integration.
4.4.1 Honghu system installation
The installation of the Honghu system is very simple. Basically, it can be done step by step according to the official documents. The system memory only needs a limit of 8G. In today's era when the memory of mobile phones exceeds 8G, it is still a very conscientious configuration requirement.
However, because Honghu uses vector computing technology, the official document suggests that Honghu Computing Engine uses the AVX2 advanced instruction set for vector computing acceleration, so Honghu needs to run on a CPU that supports the AVX2 instruction set.
Of course, the official document gives the test method, you can run the following command to know whether the CPU supports the AVX2 instruction set:
It just so happened that I had an old server ready-made, so I quickly tested it. Unfortunately, he's...not supportive.
Soon, I found a cloud host and solved it perfectly.
The installation needs to install docker and some other related components, but the installation process is very simple, just follow the official installation document ( https://yanhuang.yuque.com/staff-sbytbc/rb5rur/auwfm3? ) step by step.
This is the interface after the installation is complete, it is indeed "simple from big numbers".
4.4.2 Receive MQTT data
In order to quickly try the functions of Honghu, we need to fill in the data first. Of course, if it is just a trial, you can import data with csv, json and other files.
Since in the previous architecture, we will use EMQX to collect data from various devices, so during the trial process, I simply developed a service that receives MQTT data from EMQX and forwards it to Honghu, and has published it on the Github platform superior.
The development process is not detailed, you can go to the following address to see and download:
https://github.com/neulf/mqtt-honghu
4.4.3 Data precomputation technology
Due to the increase in the amount of data, we generally use Rollup for pre-computation on InfluxDB to enhance the ability to process large data and improve query performance. Therefore, for a new big data product, naturally, we are also very concerned about data precomputation.
From the official documents, we found that Honghu provides a variety of data pre-computing technologies, including materialized views, pre-stored queries, and scheduled reports. You can see the application scenarios of these technologies in the official documents, so I will not repeat them here. .
4.4.4 Flexible query
Bird & Bird provides simple query and advanced query functions, and supports SQL query, a common productivity tool in the industry, which is very convenient for most engineers, data analysts and even some senior operators.
For an introduction to Honghu SQL, please refer to here
（https://yanhuang.yuque.com/staff-sbytbc/rb5rur/znakv0?）
4.4.5 Rich out-of-the-box charts
Honghu provides a very rich chart. For a person who has used some commercial BI products (such as: Tableau, PowerBI), this community product is really powerful. We try to integrate the functions that need to be customized before, such as The comparison curve of indicators such as AGC command and system output power is realized by Honghu's advanced query function, which is really very simple.
4.5 Summary
Judging from the trial, Honghu is a community product that is very easy to install and has rich chart analysis functions. Since I haven't done a benchmark test on Honghu and some time-series databases in terms of insertion performance, but from the comparison of other search engines and time-series databases and personal experience, the new shared energy storage SCADA system that does not have too strict requirements for acquisition concurrency performance, Honghu can be directly used as the main storage; if it is a transformation of an existing project or has very high performance requirements, you can consider using a sequential database to cooperate with Honghu in terms of architecture.