There are a lot of big data processing tools, undoubtedly the most popular Hadoop system. Hadoop eco including HDFS, HBase, Hive, YARN, Storm, Spark, Zookeeper and other tools. The entire Big Data platform there are often Kafka, Redis and similar message queuing, caching software. These software better solution of the general big data problems, but things, the car data networking, Internet and other industrial scene has its uniqueness, if full advantage of these unique features, you can launch a proprietary big data processing platform of Things , orders of magnitude improve data processing capabilities, and reduce development and operational costs. Just open source TDengine is one such product.
TDengine designed for networking, car networking and other data series large space design, its core function is a sequence database. But to reduce the complexity of development and maintenance of large data operation platform, further reducing the computational resources, TDengine also provide large data processing required for the message queue, the message subscription, buffer, flow calculation. TDengine advantages are obvious, mainly in the following areas:
Significantly improve query performance and data insertion
Things data is structured, so TDengine adopted a structured storage, rather than the popular KV storage. Things scene, each data collection point is the only source of data, the data is timing, and user data are often concerned a period of time, rather than a particular point in time. Based on these characteristics, TDengine individual requirements of each acquisition device built form. If there are 10 million devices, we need to build 10 million table.
Based on this, the data collected by any device in the storage medium may be stored in a continuous one, and ordered by time. Thus a single device query data period, the number of stages there query performance improvement. On the other hand, although different devices depending on the network, the time to reach the server can not control, it is completely out of order, but for the same device, the timing of data points is guaranteed. A device a table, a table ensures that the inserted data sequence is guaranteed, so that the data inserting operation becomes a simple adding operation, can be greatly improved insertability.
KV benefits of storage is not defined in the database table structure, each record can transform format. But things, car networking these scenarios, the general data format is fixed, changes in the frequency of low and TDengine achieve an efficient way to modify table structure, so TDengine take to format the memory will not bring too much inconvenient.
Significantly reducing the cost of hardware or cloud service
Since the data insertion query performance greatly enhance the computing resources required for the system is greatly diminished. On the other hand, things acquired physical quantity is changing with time, but under normal circumstances, is gradual, so TDengine take storage column, the same physical quantity at a plurality of points of time acquired continuously stored, this will exponentially increasing the compression efficiency. Further TDengine take different data types for different compression methods, such as delta-delta encoding, simple 8B methods, zig-zag and the like, so that further improve the compression ratio. Compared with the common database, it has been tested Things scene, TDengine storage space is less than 1/5, significant savings in storage resources.
Greatly simplifying big data system architecture
Internet applications and is not the same, the scene of things, as long as the specified number of networked devices, the data acquisition frequency, the flow rate required by the system is more accurately estimated, unlike bis 11, the electricity supplier can flow several times change, and things flow is relatively stable. At the same time, networking equipment have some data buffering capability to prevent the network connection fails, and therefore the demand for Internet of Things platform message queue is not so strong. Internal TDengine implements a simple message queue, while providing subscription feature, so that no use of the message queue Kafka and similar software.
TDengine database assigned a fixed memory area, the new data is inserted, to be written to memory. FIFO memory in accordance with the principles of management, insufficient memory, the old data will be persistent storage, while the memory of the old data will be overwritten by the latest. TDengine also ensures that any one device must last record in memory, if the application you want to get the latest data or status of each device, will have direct access to memory, this design allows the system may no longer need this type of software Redis .
Things are a data stream of data, based on sliding window, TDengine query pulled back timing is calculated, there is provided a simplified flow calculations may be made of a variety of real-time statistics aggregation operations, such things for general scenarios, longer Spark need to use other types of flow calculation software.
TDengine thus provides a large database of data required for processing, caching, message queues, series flow calculation functions. Use TDengine, in things big data platform completely discarded Kafka, HDFS, HBase, Spark, Redis and other software, big data platform greatly simplifies design, reduces large research and development costs, and the system will be more robust, more data consistency guaranteed.
Powerful historical data analysis capabilities
Allowing users to process historical data and real-time data is completely transparent, does not distinguish between historical data and real-time data on TDengine design. Users only need to specify the SQL statement in the period, TDengine automatically decide whether, or to obtain data from the network storage from memory from the local hard disk, to achieve such applications becomes simple.
Each data storage device in blocks, and each block has been done prepolymerization (and such, maximum, minimum, etc.), such a device performing a statistical operation time of each segment, it is possible to scan the raw data without It can be calculated for improved performance. Even if some computing needs to scan the raw data, but because of a piece of data is stored contiguously, the read speed far exceeding the common database, calculation and analysis speed is increased dramatically. And because the structured storage, unpacked, do not do any parsing, read into memory can be directly calculated with respect to the NoSQL database, calculation and analysis speed is increased dramatically.
TDengine defines a new concept of super table to describe the same type of equipment. After each device to a static or table marked tag, the tag value can be used to screen out devices that meet the filter criteria, then the data portion of this polymerization apparatus. TDengine also designed a special mechanism, a plurality of devices for data aggregation, data only needs to scan a document, so that significantly reduce the number of IO operations, improve calculation speed of polymerization. To improve ease of use, the user can own TDengine the shell, or Python, R, Matlab various query tools such as direct or analysis of Ad Hoc. TDengine used to do things, car networking, the Internet industry data warehouse, would be an ideal choice.
Zero operation and maintenance management, zero learning costs
TDengine installation package is less than 2M, download, install a few seconds to get. For Enterprise Edition, the machine will be able to complete a command added to the cluster, but the database is automatically backed up in real time, without having to manually sub-library sub-table, operation and maintenance extremely simple. The system uses standard SQL, support for C / C ++, Java, Python , Go and other language development interface, support for JDBC, support for RESTful interface. Use it as if using MySQL, almost no learning costs.
Seamless integration with third-party tools
Currently TDengine data acquisition side, have support Telegraf, Kafka, follow-up will also support MQTT, OPC and the like. On the application side, it has support Grafana visualization tools, support Matlab, R and some BI tools. Because TDengine support JDBC interface, the interface is easy to implement and third-party tools, can be predicted that more tools will be seamlessly integrated.
For operation and maintenance monitoring scene, without writing any code, as long as the open source Telegraf, Grafana and TDengine configured, you can quickly build an efficient operation and maintenance monitoring platform.
Open source
TDengine developed by the Beijing Tao Si Data Technology Co., Ltd., did not rely on any third-party software. Development time has been more than two years, and has paid a number of business customers involved in electric power, machine tools, smart city, vehicles and other networks, the use of customer feedback is very good. The good news is, think of Tao data TDengine core storage, calculation engine completely open source. Community Edition TDengine of things can meet certain scale, vehicle networking, industrial applications of the Internet. Because Tao think the data core team in Beijing, compared to other open source software, software engineer should be able to give China to provide better local services.
Epilogue
TDengine on ease of use, functionality, timing has been far more than the other database performance. Using TDengine, networking, the Internet industry, to build a big data platform operation and maintenance monitoring becomes extremely simple, with superior performance, not only reduce hardware costs, operation and maintenance costs, but also significantly reduce the need for research and development and operation and maintenance personnel.
Because open source is free, and the installation package is less than 2M, you may wish to visit tdengine.com download a try.
Click on "read the original" download open source TDengine
