Observation Cloud achieves 10 times cost-effective improvement in log storage and analysis｜SelectDB technical team

Author: Observation Cloud CEO Jiang Shuomiao & Feilun Technology Technical Team

As cloud computing matures, more and more companies are migrating their businesses to the cloud. Traditional monitoring and troubleshooting methods can no longer meet the needs of companies. The observation cloud can provide overall data analysis, insight, visualization, automation, monitoring and alarming, intelligent inspection, safety inspection and other services. In order to better provide the above services, the observation cloud is required to be able to uniformly integrate massive data from multiple scenarios and structures, and provide comprehensive log retrieval and analysis capabilities to quickly implement data query, filtering and analysis.

Therefore, Feilun Technology has fully cooperated with Observation Cloud. Through SelectDB’s inverted index capability, Variant data type, hot and cold data tiered storage and other features, it injects strong power into observation cloud log storage and analysis scenario services, achieving a 70% reduction in storage costs. At the same time, the query performance is improved by 2-4 times, and the overall cost-effectiveness is finally improved by 10 times!

GuanceDB original architecture

Observation Cloud has strong data access capabilities. Through the self-developed All In One collection tool DataKit, it can obtain data from different layers such as different terminals, business layers, middleware, and infrastructure, and perform preprocessing and meta-information association at the same time. In addition to extensive support for log data, Datakit also supports the collection and processing of infrastructure time series indicators, link tracking, security events, and user behavior data on the APP or browser side. In order to meet the diverse needs of multiple scenarios, DataKit is not only fully compatible with open source probes and collectors, but also supports access to data sources in custom formats.

The data collected by DataKit will be uniformly stored in GuanceDB after being processed by the core computing layer. GuanceDB is a multi-modal database composed of a variety of database technologies independently developed by Observation Cloud.

The internal architecture of GuanceDB is shown in the figure above, which mainly consists of two layers: Query Engine and Storage Engine. In terms of logical structure, the query engine and the storage engine are abstractly decoupled, and the overall architecture is pluggable and replaceable.

Observation Cloud developed the time series storage engine Metric Store based on the VictoriaMetrics storage module, and also integrated Elasticsearch/OpenSearch in the pan-log scenario. This design enables GuanceDB to have a unified external writing and query interface, which can adapt to different types of data formats and business needs. In its current implementation, MetricStore already has excellent performance. However, for the processing of log and user behavior data, Elasticsearch has many shortcomings. The specific manifestations are as follows:

• Writing takes up a lot of resources: Elasticsearch will occupy high CPU and memory resources when processing high-frequency writing of large amounts of data, which will not only significantly increase cluster costs. , and it will also occupy the resources occupied by the query.
• Poor support for schemaless tables: Elasticsearch has limited support for Schemaless. The current Dynamic Mapping will frequently cause field type conflicts when faced with a large number of user-defined fields, resulting in Data loss requires manual intervention for manual processing.
• Poor aggregation query performance: Elasticsearch has poor aggregation performance when faced with massive data. For example, when calculating quantiles and error rates for billions of data, timeouts can easily occur, making it difficult to meet business analysis needs for large-scale data.

Selection goals and research

The problems existing in Elasticsearch in the original architecture drove us to upgrade the architecture. Before upgrading, we investigated a variety of databases including SelectDB. Combined with actual observability scenarios, our selection goals are as follows:

• High throughput and high performance: In observable scenarios, the scale of business data will increase linearly with business complexity and business scale. To meet this demand, we need a storage solution that can support both high-throughput real-time writing and high-performance data analysis. At the same time, the cluster itself is easy to operate and maintain and supports horizontal expansion.
• Full-text inverted index: Full-text inverted index can significantly improve retrieval performance and reduce query resource overhead, and is an essential capability for efficient log analysis. During the research, we also noticed index-less solutions like Loki. Although this type of solution is simple, when the request QPS is slightly high, the competition for disk IO and CPU resource overhead during full disk scanning will be very fierce, and it cannot carry log charts. Display, cluster screening analysis, real-time alarm and other business needs.
• Supports multiple business writing and query scenarios: The business scenarios collected by the observation cloud are rich and diverse, including massive throughput of append writing, and the overall periodicity of object data such as processes and hosts. Updates, session partial updates in RUM scenarios, etc., while covering multiple query scenarios such as high-frequency enumeration, list query, large-scale aggregation query, etc. This requires a new solution that can support multiple business writes and diversified scenario queries.
• Support schemaless tables: There is a large amount of field meta-information in observable business scenarios that are manually maintained by business engineers based on business needs. In order to better adapt to this scenario, The storage layer needs to support SchemaLess, which eliminates the need for the upper-layer business to maintain the Schema information of the data table, and can automatically handle type conflicts.
• Support large-scale tenant isolation: In a SaaS scenario, we have a large number of tenants and sub-tables, and these metadata themselves will cause great management pressure on the system. When using Elasticsearch, a single cluster can support a limited number of indexes. Once a certain number of indexes is reached, performance will drop sharply, so data needs to be dispersed into different clusters, which causes a lot of trouble in cluster management
• Reduce long-term storage costs: The value of observable data will decrease over time. We hope to reduce long-term storage through technical means such as hot and cold separation, storage and calculation separation, etc. of data to object storage to reduce the overall data storage cost.

Taken together, SelectDB can meet most of the requirements for observation clouds and performs well in comparison with similar products. We will also introduce the transformation practice based on SelectDB in detail in the following chapters. It is particularly worth mentioning that in the preliminary research, the following features were important features that attracted us:

• SelectDB inverted index can save storage space by more than 80%, write speed is 5 times that of Elasticsearch, and query performance is 2.3 times that of Elasticsearch.
• SelectDB has designed the Variant data type for semi-structured data such as JSON. JSON of any structure can be stored in the Variant type. It can automatically analyze the fields and types inside JSON and use columnar storage for frequently appearing fields to improve storage and The effectiveness of analysis
• SelectDB can support thousands of databases and tens of thousands of data tables, enabling one tenant to use one database independently, achieving multi-tenant data isolation requirements and meeting data isolation and security.

Storage architecture upgrade based on SelectDB

Therefore, we introduce SelectDB to upgrade the internal architecture of GuanceDB. In order to better introduce how SelectDB functions as a storage engine in GunaceDB, we first introduce the DQL query language.

In observability scenarios, almost all queries involve time filtering, and most aggregations also need to be performed according to time windows. For time series, it is also necessary to support rollup before and after the time window on a single sequence. In these scenarios, using SQL to express the same semantics requires nesting multiple levels of subqueries, making the expression process and writing extremely complex.

Therefore, we tried to simplify the syntax elements, and based on this, we designed a new query language DQL, and enhanced common calculation functions in observable scenarios. All indicators, logs, link tracking, objects, etc. can be queried through DQL. Observable data.

From the internal structure of GuanceDB, we used SelectDB to replace Elasticsearch/OpenSearch in this upgrade, and the original query architecture remains unchanged.

Next, we introduce how DQL queries work after introducing SelectDB:

As shown in the figure above, Guance-Insert is the data writing component, and Guance-Select is the DQL query engine.

• In Guance-Insert, the data batch storage logic of each tenant is implemented, the two major indicators of write throughput and write latency are balanced, and the data is written to the SelectDB Doris BE component through the Stream Load interface as efficiently as possible. When a large amount of logs are generated, this method can save batches very quickly, and the average log storage delay is 2-3 seconds.
• In Guance-Select, Guance-Select will choose whether to push the query down to FE calculation based on the SQL support of the current query SelectDB. Normally, common aggregation queries can be pushed down to FE for calculation, but when encountering SQL semantics or functions that FE does not support, we will choose Fallback to only push down the predicate to BE, and obtain the Arrow format through the Thrift RPC interface. The data is stored in the column and then calculated in Guance-Select. Since this solution cannot push the calculation logic down to BE, the actual performance will be slightly worse than the query in FE. However, in most scenarios, this solution can meet the needs.

The current query architecture is a hybrid computing architecture that integrates FE and BE capabilities. DQL can not only take advantage of SelectDB's fully optimized query capabilities, but also allow syntax expansion without being limited by SelectDB's own SQL capabilities.

Benefits from architecture upgrades

01 Reduce storage costs by approximately 70% and increase query performance by 3 times

The introduction of SelectDB has achieved a significant reduction in overall costs. Previously, we used an Elasticsearch cluster composed of 20 16C 64G cloud hosts in an availability zone on the cloud to provide query services, and also used an independent index writing service (equivalent to using 20 cloud hosts). After replacing with SelectDB, only 13 cloud hosts with the same configuration are needed, and the total cost is reduced by 67%. The significant cost reduction is mainly due to two factors:

• SelectDB has higher write performance than Elasticsearch: When dealing with sustained high-throughput writes of 1GB/s, the CPU occupied by SelectDB remains below 20%, equivalent to the cost of approximately 2.6 cloud hosts, which is only the cost of the Elasticsearch index write service. 13%. This advantage can reduce writing costs while handling larger burst traffic and ensuring system stability.

• The compression rate of SelectDB data and index is higher than that of Elasticsearch: SelectDB data and index use column storage and ZSTD compression technology, making the overall compression ratio of the online cluster up to 1:8, while the compression ratio of Elasticsearch is only 1:1.5, so when using SelectDB, The storage space occupied is only about 20% of Elasticsearch.
• SelectDB supports hierarchical storage of hot and cold data: we can store hot data that has been recently queried frequently on the local disk, and cold data that has not been used for a long time is automatically uploaded to object storage, which can significantly reduce data storage costs. At the same time, SelectDB supports automatic hot and cold data migration based on storage policy configuration, and data lifecycle management and query are transparent to upper-layer applications, making it more flexible and convenient to use. In addition, SelectDB can also accelerate access to cold data through local Cache, thereby improving the user experience of querying cold data.

The introduction of SelectDB has achieved significant improvement in query performance. After reducing the number of machines, we compared the performance of the same query in the two clusters. Practice shows that SelectDB's point query and list query speed are nearly 2 times faster than Elasticsearch. When aggregation queries are not sampled, SelectDB is faster than Elasticsearch. Elasticsearch is nearly 4 times faster.

To sum up, after using SelectDB to replace Elasticsearch, it only uses 1/3 of the cost of Elasticsearch and obtains 2 to 4 times performance improvement. The overall cost-effectiveness is improved by nearly 10 times!

02 Inverted index meets the needs of full-text retrieval in log scenarios

The inverted index can significantly improve the performance of full-text retrieval and reduce the resource overhead of queries. It is an essential capability for efficient log analysis. SelectDB supports inverted indexes. The following is an introduction to the key capabilities in our migration from Elasticsearch to SelectDB:

• Supports string full-text search, including matching multiple keywords at the same timeMATCH_ALL, matching any keywordMATCH_ANY, and matching phrases for querying, which can completely cover the original functions on Elasticsearch. MATCH_PHRASE query method. When we create an inverted index for log text content, we use MATCH_PHRASE
• Supports English, Chinese and Unicode multi-language word segmentation, and Chinese word segmentation also supports custom lexicon and custom stop words. We configured the Chinese lexicon and stop words originally used on Elasticsearch to SelectDB, completing a smooth migration of user experience.
• Accelerates common equal values ​​(=, !=, IN) and range queries (>, >=, <, <=), and supports numbers, dates, and string types.

CREATE TABLE httplog
(
  `ts` DATETIME,
  `clientip` VARCHAR(20),
  `request` TEXT,
  INDEX idx_ip (`clientip`) USING INVERTED, --不分词
  INDEX idx_req (`request`) USING INVERTED PROPERTIES("parser" = "chinese") --中文分词
)
DUPLICATE KEY(`ts`)
...

-- 查询clientip为'8.8.8.8'的最新10条数据
SELECT * FROM httplog WHERE clientip = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
-- 检索request字段中有error或者404的最新10条数据
SELECT * FROM httplog WHERE request MATCH_ANY 'error 404' ORDER BY ts DESC LIMIT 10;
-- 检索request字段中有image和faq的最新10条数据
SELECT * FROM httplog WHERE request MATCH_ALL 'image faq' ORDER BY ts DESC LIMIT 10;
-- 检索request字段中有'查询错误'词组的最新10条数据
SELECT * FROM httplog WHERE request MATCH_PHRASE '查询错误' ORDER BY ts DESC LIMIT 10;

In addition to functionally meeting the needs of full-text log retrieval, SelectDB inverted index alsosupports online on-demand index increase and decrease. Elasticsearch indexes are fixed when they are created, and index fields cannot be added later. This requires planning in advance which fields need to be indexed. If you need to change the index later, you need to rewrite it, and the cost of change is very high. SelectDB supports adding indexes on demand during operation, and newly written data indexes take effect immediately. At the same time, SelectDB can control which partitions are indexed, making it very flexible to use.

03 Variant data type solves the pain point of frequent changes in data schema

In observable scenarios, data are diverse and change frequently. For example, when we collect every click and import of users on a web page, we may add new business indicators. Such a scenario puts forward higher requirements for the real-time performance of database Schema changes.

In common databases, the schemas of most data tables are static, but some databases such as Elasticsearch can implement dynamic schemas through Mapping. However, dynamic schemas may encounter field type conflicts or the number of fields reaching the upper limit due to historical fields not being invalidated. After the introduction of SelectDB, we use the latest feature Variant dynamic data type (this feature will be officially released in Apache Doris 2.1 version) to support such scenarios well.

SelectDB has designed the Variant data type for semi-structured data, which has the following special capabilities:

• Supports any legal JSON data to be stored in Variant type columns, and can automatically identify subfields and types in JSON.
• The Variant data type can avoid the Schema explosion problem caused by too many fields. For frequently occurring subfields, the Variant type uses columnar storage to improve the efficiency of data storage and analysis. For subfields that appear infrequently, the Variant type will merge them into one column for storage to avoid the number of columns being too large.
• The Variant data type can avoid the problem of business change field type conflicts and inability to write. Variant allows a field to have different types and use different storage methods. New and old data are stored in different types, and the mixed part of alternating old and new data is stored in the minimum common type.

The most significant difference between the Variant type and Elasticsearch Dynamic Mapping is that the scope of Dynamic Mapping is the complete life cycle of the current table, while the scope of Variant is only within the current dynamic partition. This differentiated design allows Variant columns to expire periodically as business data writing changes, thereby reducing the probability of type conflicts. For example, when we change the business logic code today and rename some business fields, the old field names will not appear in the Variant column tomorrow. Therefore, we can think that Variant only maintains the type data of the latest data.

In additionWhen field types conflict within a single partition, it will be upgraded to the JSON data type, thereby avoiding data errors and data loss . For example, the status field is used in two places in the business system, one of which is a string and the other is a number. Then when we query, we can choose what the current query needs according to the actual semantics. String, number, or both. (Suppose you write status = "ok" in the filter condition, then only data of type status that is string will be filtered.)

After using the Variant data type, users do not need to be aware of the existence of Variant during actual writing and querying. Users can add or delete fields according to their own business needs, just like using ordinary columns. When querying, there is no need for additional syntax or annotations, just treat it as an ordinary column and perform operations.

In the current version, the Variant data type requires additional type assertions when used, and automatic type assertions will be updated in subsequent versions. Currently in DQL queries, we have implemented automatic type assertion for Variant columns. In most cases, assertions can be made directly based on the actual data type of the Variant. Only in rare cases of type conflicts, the Variant column will be upgraded to the JSON data type. At this time, we will use the aggregation operator or operator association in the DQL query. semantics to make the actual assertion.

04 Design sampling logic to accelerate aggregate query performance

During the adaptation process, we found that although SelectDB has powerful performance, it still takes up a lot of system resources when aggregation calculations of very large data sets are required, and the calculation overhead is relatively high. In observable scenarios, most calculations are qualitative analysis rather than quantitative absolute value precise analysis.

Based on this business background, we designed the following sampling logic in GunaceDB:

• Estimate the number of original data rows within the query time range, start sampling when the number of original data rows to be queried is greater than 10 million, and fix the number of sampled rows at 10 million to calculate the sampling rate in reverse.
• At the storage layer, SelectDB's TableSample capability is used for actual data sampling, and a certain table write balancing strategy is used to ensure that the aggregation results do not produce serious deviations.
• At the query engine layer, the sampling results are adapted according to different aggregation operators. Most calculations such as quantiles and averages do not need to be processed. Only the Sum and Count functions need to be processed in equal proportions.
• When the Count aggregation query hits too few results after sampling, we will turn off sampling and query again to avoid large errors. In addition, we will mark the sampling rate in the response result. When the user suspects that the sampling result is biased, he can turn off sampling and re-initiate the request.

Under such sampling logic, the computing overhead and user waiting time required for ultra-large-scale calculations can be significantly reduced. Compared with the previous performance, the performance is improved dozens of times, greatly improving the user experience.

Conclusion

The introduction of SelectDB meets our Schema Free requirements and solves the pain point of frequent changes in data schema; improves the performance of data writing, ensures the timeliness of data writing and real-time querying; improves the performance of full-text retrieval and reduces the query time Resource overhead... All in all, the application of SelectDB enabled the observation cloud to reduce storage costs by 70%, improve query performance by 2-4 times, and ultimately achieve a 10-fold improvement in overall cost performance! In the future, we will continue to collaborate with Feilun Technology to create solutions that are more popular with users, and we will also jointly promote the development and growth of the Apache Doris community!