TDengine assists SF Technology in the transformation of big data monitoring

Author: Yin Fei

Small T introduction : SF Technology big data cluster needs to collect massive amounts of monitoring data every day to ensure the stable operation of the cluster. Although OpenTSDB+HBase was used as the storage solution for the full monitoring data of the big data monitoring platform, there are many pain points, and the full monitoring data storage solution must be transformed. Through research on time series data storage solutions such as IoTDB, Druid, ClickHouse, and TDengine, we finally chose TDengine. After the big data monitoring platform adopts TDengine , the stability, writing performance and query performance have been greatly improved, and the storage cost has been reduced to 1/10 of the original solution.

Scenarios and Pain Points

SF Technology is committed to building a smart brain and building smart logistics services. It continues to develop in the fields of big data and products, artificial intelligence and applications, and comprehensive logistics solutions, and is in a leading position in China's logistics technology industry. In order to ensure the smooth operation of various big data services, we have built a big data monitoring platform around OpenFalcon. Since OpenFalcon itself uses rrdtool as data storage, it is not suitable for the storage of full monitoring data, so we use OpenTSDB+HBase as the storage solution for full monitoring data of the big data monitoring platform.

Currently, the entire platform writes billions of entries per day on average. With the increasing amount of data accessed by the big data monitoring platform, we have many pain points to solve, including many dependencies, high usage costs, and unsatisfactory performance.

Many dependencies and poor stability : As the underlying big data monitoring platform, it relies on big data components such as Kafka, Spark, and HBase for data storage. Too long data processing links will reduce the reliability of the platform. At the same time, because the platform relies on big data components, and the monitoring of big data components depends on the monitoring platform, when the big data components are unavailable, the monitoring platform cannot solve the problem in time. to locate.
High cost of use : Due to the huge amount of monitoring data written, and the full amount of monitoring data needs to be saved for more than half a year to trace the problem. Therefore, according to the capacity planning, we use 4-node OpenTSDB+21-node HBase as the full monitoring data storage cluster. After compression, about 1.5T (3 copies) of storage space is still required per day, and the overall cost is relatively high.
Performance can not meet the requirements : OpenTSDB, as a full-scale monitoring data storage solution, basically meets the requirements in terms of writing performance, but cannot meet the requirements in terms of daily large-span and high-frequency queries. On the one hand, OpenTSDB query returns slow results, and takes more than ten seconds when the time span is relatively large; on the other hand, the QPS supported by OpenTSDB is low. use.

Technical selection

In order to solve the above problems, it is necessary to upgrade the full monitoring data storage solution. In terms of database selection, we have pre-researched and analyzed the following databases:

IoTDB : The newly hatched Apache top-level project, contributed by Tsinghua University, has good single-machine performance, but we found that it does not support cluster mode during our research, and the single-machine mode cannot meet the needs of disaster recovery and expansion.
Druid : Powerful, scalable distributed system, self-healing, self-balancing, and easy to operate, but relying on ZooKeeper and Hadoop as deep storage, the overall complexity is high.
ClickHouse : The performance is the best, but the operation and maintenance cost is too high, the expansion is very complicated, and the resources used are more.
TDengine : The performance, cost, and difficulty of operation and maintenance are all satisfied, and it supports horizontal expansion and high availability.

Through comprehensive comparison, we initially selected TDengine as the monitoring data storage solution. TDengine supports a variety of data import methods, including JDBC and HTTP modes, which are more convenient to use. Due to the high performance requirements for monitoring data writing, we finally adopted the Go Connector. The access process requires the following operations:

Data cleaning to remove data with incorrect format;
Data formatting, converting data into entity objects;
SQL statement splicing, judging the data, and determining the written SQL statement;
Write data in batches. In order to improve the writing efficiency, after the SQL splicing of a single piece of data is completed, the data is written in batches.

data modeling

Before accessing data, TDengine needs to design the schema according to the characteristics of the data to achieve the best performance. The data characteristics of the big data monitoring platform are as follows:

The data format is fixed, with its own timestamp;
The content of the uploaded data is unpredictable, and new nodes or services will upload new label content, which makes the data model unable to be created uniformly in the early stage, and needs to be created in real time according to the data;
There are not many data label columns, but the label content changes a lot; the data value columns are relatively fixed, including timestamp, monitoring value and sampling frequency;
The amount of data for a single piece of data is small, about 100 bytes;
A large amount of data per day, more than 5 billion;
Keep for more than 6 months.

According to the above characteristics, we constructed the following data model.

According to the data model suggested by TDengine, each type of data collection point needs to establish a super table, such as disk utilization. If the disk utilization on each host can be collected, it can be abstracted into a super table. Combined with our data characteristics and usage scenarios, create a data model as follows:

Using indicators as a super table is convenient for aggregate analysis and calculation of the same type of data;
The monitoring data itself includes tag information, and the tag information is directly used as the tag column of the super table, and the same tag value forms a sub-table.

The library structure is as follows:

The super table structure is as follows:

Implementation

The big data monitoring platform is the base for the stable operation of the upper-level big data platform, and it is necessary to ensure the high availability of the entire system; as the business volume increases, the monitoring data volume continues to grow, and it is necessary to ensure that the storage system can be easily scaled horizontally. Based on the above two points, the overall architecture of TDengine landing is as follows:

In order to ensure the high availability and scalability of the entire system, we use nginx cluster for load balancing at the front end to ensure high availability; separate the client layer to facilitate expansion and contraction according to traffic requirements.

The implementation difficulties are as follows.

Data writing : Since the upload interface of monitoring indicators is open, only the format is verified, and the written data indicators are uncertain, so the super table and sub-table cannot be created in advance. In this way, it is necessary to check whether a new super table needs to be created for each piece of data. If you need to access TDengine every time you judge, it will cause the writing speed to drop sharply and completely fail to meet the requirements. In order to solve this problem, a cache is established locally, so that only one query of TDengine is required, and subsequent write data of related indicators can be directly written in batches, which greatly improves the write speed. In addition, the speed of batch creation of tables in versions before 2.0.10.0 is very slow. In order to ensure the writing speed, it is necessary to insert the table and insert data in batches, and the data information of the sub-table needs to be cached. The later version optimizes the sub-table creation function , the speed has been greatly improved, and the data insertion process has also been simplified.
Query questions : 1. Query bugs. The monitoring platform data is mainly displayed through Grafana, but in the process of using it, we found that the officially provided plug-in does not support parameter settings. According to our own needs, we modified it and provided it to the community. In addition, during use, a serious query bug was triggered: when more kanban boards are set, refreshing the page will cause the server to crash. After investigation, it was found that when a dashboard in Grafana was refreshed, multiple query requests would be initiated at the same time to process concurrent queries, which was later officially repaired. 2. Query a single point of question. TDengine's native HTTP query is completed by directly querying a specific server. This is risky in a production environment. First, all queries are concentrated on one server, which easily overloads a single machine; in addition, the high availability of query services cannot be guaranteed. Based on the above two points, we use Nginx cluster as a reverse proxy in front of TDengine cluster to evenly distribute query requests on each node, which can theoretically expand query performance infinitely.
Capacity planning : The data type and data scale have a great impact on the performance of TDengine. It is best to plan capacity for each scenario based on its own characteristics. The influencing factors include the number of tables, data length, number of replicas, and table activity. Adjust configuration parameters based on these factors to ensure optimal performance, such as blocks, caches, ratioOfQueryCores, etc. According to the communication with Taosi engineers, the capacity planning calculation model of TDengine was determined. The difficulty of TDengine capacity planning lies in memory planning. In general, a three-node 256G memory cluster supports a maximum number of sub-tables of about 2000w. If it continues to increase, the writing speed will decrease, and a part of the memory space needs to be reserved as a query. Cache use, generally keep about 10G. If the number of child tables exceeds 2000w, you can choose to expand new nodes to share the pressure.

Transformation effect

After the transformation, the TDengine cluster can easily handle the writing of full monitoring data, and is currently running stably. The structure diagram after the transformation is as follows:

In terms of stability : After the transformation, the big data monitoring platform gets rid of the dependence on big data components, effectively shortening the data processing link. Since its launch, it has been running stably, and we will continue to observe it in the future.
Write performance : The write performance of TDengine has a lot to do with the write data. After adjusting the relevant parameters according to the capacity planning, under ideal conditions, the cluster write speed can reach a maximum write speed of 90w/s. Under normal circumstances (there are new tables, mixed inserts), the write speed is 20w/s.
Query performance : In terms of query performance, when using the precomputed function, the query p99 is within 0.7 seconds, which can already meet most of our daily query needs; in the case of large-span (6 months) non-precomputed queries , the first query takes about ten seconds, and the subsequent similar queries will take a significant reduction (2-3s). The main reason is that TDengine will cache the latest query results. Similar queries first read the existing cached data, and then combine the newly added data. do aggregation.
In terms of cost : the number of physical servers on the server side is reduced from 21 to 3, and the daily storage space required is 93G (2 copies), which is only about 1/10 of that of OpenTSDB+HBase under the same copy, which is relatively versatile in reducing costs. Data platforms have great advantages.

Summarize

At present, from the perspective of big data monitoring, TDengine has great advantages in cost, performance and convenience of use, especially in terms of cost. In the process of pre-research and project implementation, the engineers of Taosi provided professional and timely help, and I would like to express my gratitude here. It is hoped that TDengine can continue to improve performance and stability, develop new features, and we will also conduct secondary development according to our own needs and contribute code to the community. I wish TDengine better and better. For TDengine, we also have some feature points that we look forward to improving:

More friendly to table name support;
Support for other big data platforms, joint query requirements;
Support richer SQL statements;
Grayscale smooth upgrade;
Sub-table automatic cleaning function;
The recovery speed of abnormal cluster downtime.

In the future, we will also try to apply TDengine in more scenarios of SF Technology, including:

The IoT platform, as the underlying IoT data storage engine, builds the SF Express Big Data IoT platform;
Hive on TDengine realizes joint query with other existing data sources through Hive on TDengine, so that it can be smoothly used with existing systems and reduce the access threshold.