OpenTSDB is a classic time series database system. It does not develop its own storage engine, but is based on HBase . For enterprises that already have HBase basic services, the threshold is lowered. And thanks to its first-mover advantage, OpenTSDB has many applications in the field of operation and maintenance monitoring. However, because of the dependence on HBase, the performance and compression efficiency of the system have gradually become the bottleneck. With the expansion of business system scale, the problems of deployment cost and operation efficiency are becoming more and more serious. In addition, the function upgrade of OpenTSDB is relatively slow. Compared with it, TDengine has obvious advantages:
- The performance of data writing and querying far exceeds that of OpenTSDB;
- Efficient compression mechanism for time series data, the storage space on disk after compression is less than 1/5 of OpenTSDB;
- The installation and deployment is very simple. A single installation package completes the installation and deployment. Except for the taosAdapter which needs to rely on the Go operating environment, it does not rely on other third-party software. The entire installation and deployment is very fast;
- The built-in functions provided cover all query functions supported by OpenTSDB, and also support more time series data query functions, scalar functions and aggregation functions, and support multiple time window aggregations, join queries, expression operations, multiple grouping aggregations, user-defined Advanced query capabilities such as sorting, and user-defined functions. Using SQL-like grammar rules, it is easier and easier to learn, and there is basically no learning cost.
- Support up to 128 tags, the total length of tags can reach 16KB;
- In addition to HTTP, it also provides interfaces in Java, Python, C, Rust, Go and other languages.
If we migrate the applications originally running on OpenTSDB to TDengine, it can not only effectively reduce the occupation of computing and storage resources, reduce the scale of deployment servers, but also greatly reduce operation and maintenance costs, making operation and maintenance management work simpler and easier. Easily and dramatically reduce total cost of ownership.
This article will use "the most typical and widely used operation and maintenance monitoring scenario" to illustrate how to quickly, safely and reliably migrate OpenTSDB-based applications to TDengine without writing a single line of code.
1. Typical operation and maintenance monitoring application scenarios
The overall system architecture of a typical operation and maintenance monitoring application scenario is shown in the following figure (Figure 1).
Figure 1. Typical architecture of operation and maintenance monitoring scenarios
In this application scenario, there are Agent tools deployed in the application environment responsible for collecting machine metrics (Metrics), network metrics (Metrics) and application metrics (Metrics), data collectors that aggregate the information collected by the Agent, and responsible for data persistence Storage and management systems and monitoring data visualization tools (eg: Grafana, etc.).
Among them, the Agent deployed on the application node is responsible for providing operation indicators from different sources to collectd/Statsd, and collectd/StatsD is responsible for pushing the aggregated data to the OpenTSDB cluster system, and then using Grafana to visualize the data.
2. Migration service
-
TDengine installation and deployment
The first is the installation of TDengine. Download the latest stable version of TDengine from the official website, decompress it and run install.sh to install it. For the use help of various installation packages, please refer to "Installation and Uninstallation of Various Installation Packages of TDengine" . Note that after the installation is completed, do not start the taosd service immediately, and start it after the parameters are correctly configured.
-
Adjust data collector configuration
In TDengine 2.3 version, after the background service taosd is started, an HTTP service called taosAdapter is also automatically enabled. Using taosAdapter, it is compatible with Influxdb's Line Protocol and OpenTSDB's telnet/Json writing protocol, so we can directly push the data collected by collectd and StatsD to TDengine.
If collectd is used, modify /etc/collectd/collectd.confthe configuration file in its default location to point to the node IP address and port where taosAdapter is deployed. Assuming that the IP address of taosAdapter is 192.168.1.130 and the port is 6046, the configuration is as follows:
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "192.168.1.130"
Port "6046"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>In this way, collectd can write data to TDengine through taosAdapter. If you are using StatsD, you can adjust the configuration file accordingly.
- Adjusting the Dashborad system
After the data can be written to TDengine normally, you can adjust and adapt Grafana to visualize the data written to TDengine. There is a connection plugin for Grafana (connector/grafanaplugin) in the TDengine installation directory. It's easy to use:
First, copy the entire dist directory under the grafanaplugin directory to the Grafana plugin directory (the default address is /var/lib/grafana/plugins/), then restart Grafana, and you can see the TDengine data source under the Add Data Source menu.
In addition, TDengine also provides two sets of default Dashboard templates for users to quickly view the information saved in the TDengine library. Just import it into Grafana and activate it.
Figure 2. Importing the Grafana template
So far, we have completed the migration to replace OpenTSDB with TDengine. You can see that the whole process is very simple, no need to write code, just adjust a certain configuration file.
3. Post-migration architecture
After the migration is completed, the overall architecture of the system at this time is shown in the following figure (Figure 3). During the whole process, the collection end, the data writing end, and the monitoring and presentation end have remained stable. Except for a few configuration adjustments, no Involves any significant changes and changes.
Figure 3. System architecture after migration
The main application scenario of OpenTSDB is operation and maintenance monitoring. In this case, we can easily complete the migration to TDengine, so as to use the more powerful processing power and query performance of TDengine.
In most operation and maintenance monitoring scenarios, if you have a small-scale OpenTSDB cluster (3 nodes or less) as the storage end of monitoring data and rely on the data storage and query functions provided by OpenTSDB, you can safely Replace with TDengine and save more computing and storage resources. Under the same computing resource configuration, a single TDengine can achieve the service capabilities provided by 3 to 5 OpenTSDB nodes. If the scale is relatively large, you need to use a TDengine cluster.
If the application is particularly complex, or the application domain is not an operation and maintenance monitoring scenario, you can continue to read the next article for a more comprehensive and in-depth understanding of the advanced topics of migrating OpenTSDB applications to TDengine.