Background introduction
ByConity is suitable for a variety of business scenarios and has very good performance in real-time data access, aggregation queries of large and wide tables, complex analysis and calculations under massive data, and multi-table correlation queries. Let’s use an actual business scenario to introduce this behavior analysis system. This behavior analysis system is based on a multi-dimensional user behavior analysis platform and provides multiple analysis methods and scenarios such as event analysis, retention analysis, conversion analysis, user grouping, and user retention. This article will introduce the problems and challenges encountered by this multi-dimensional user behavior analysis platform when using the original ClickHouse cluster, and how to solve these problems and bring benefits to the business after migrating ByConity.
Figure 1
Behavior analysis system
architecture design
issues and challenges
In the early days, this system was deployed in the ClickHouse cluster. On the one hand, due to the rapid development of the business, the amount of data was increasing day by day. The maximum daily new data exceeded 320TB, the daily number of new rows exceeded 2.3 trillion, and the user data dimensions exceeded 20,000. Multiple; on the other hand, user query needs are more flexible and diverse, and they need to support detailed queries, aggregate queries, and interactive analysis queries at the same time, and provide quick response results. In addition, as the amount of data continues to increase (annual growth of 35%), we must not only be able to support the challenges brought by such a large data increase, but also control the cost growth rate within a certain range.
But it is difficult for us to do this on the existing ClickHouse cluster. The reason is that ClickHouse is based on a Shared-Nothing architecture. Each node is independent and does not share storage resources. Therefore, computing resources and storage resources are tightly coupled, which will lead to the following problems:
-
The cost of scaling up and down becomes higher and involves data migration, which prevents us from scaling up and down in real time as needed, which also leads to a waste of resources and uncontrollable costs.
-
Tightly coupled architecture will cause multi-tenants to interact with each other in a shared cluster environment, causing user queries to affect each other.
-
Since the reading and writing of nodes on the cluster are completed on the same node, reading and writing affect each other.
-
The performance support for complex queries such as multi-table join operations is not very good and cannot meet the diverse needs of users for querying.
Technology selection
Therefore, in early 2022, the business began to use ByConity with a computing and storage separation architecture as the main OLAP engine. ByConity is an open source cloud-native data warehouse that adopts a computing-storage separation architecture and supports multiple key features, such as computing-storage separation, elastic expansion and contraction, multi-tenant resource isolation, and strong consistency of data reading and writing. By utilizing mainstream OLAP engine optimizations, such as column storage, vectorized execution, MPP execution, query optimization, etc., ByConity can provide excellent read and write performance.
Figure 2
ByConity
three-layer technical architecture diagram
ByConity is an upgrade based on the open source ClickHouse architecture. It introduces an architecture that separates computing and storage. It transforms the original architecture in which computing and storage are managed locally on each node to a unified management of all tasks in the entire cluster on distributed storage. The data architecture makes each computing node a stateless pure computing node, and uses the expansion capabilities of distributed storage and the stateless characteristics of computing nodes to achieve dynamic expansion and contraction. It is precisely because of this improvement that ByConity has the following important features:
-
Resource isolation : Isolate resources for different tenants so that tenants will not be affected by each other.
-
Read and write separation : Computing resources and storage resources are decoupled to ensure that read operations and write operations do not affect each other.
-
Elastic expansion and contraction : Supports elastic expansion and contraction, and can expand and contract computing resources in real time and on demand to ensure efficient utilization of resources.
-
Strong consistency of data : Strong consistency of data reading and writing ensures that the data is always up to date and there is no inconsistency between reading and writing.
-
High performance : Adopts mainstream OLAP engine optimization, such as column storage, vectorized execution, MPP execution, query optimization, etc. to provide excellent read and write performance
business income
After we introduced ByConity, the overall performance can reach 91%, and all user queries can be completed within 10 seconds. Through feedback research from users, this performance indicator is also within the acceptable range of users. Here is a summary of the overall benefits and experience brought by our migration to ByConity:
-
Avoid resource preemption and ensure query performance is 100% stable :
On the original ClickHouse cluster, we often encountered the problem of resource crowding. Since ClickHouse did not achieve resource isolation and tenant isolation, when multiple users shared the cluster for query, when one user query resources, the overhead would be very high, which would involve The preemption of resources causes all shared user queries on this cluster to be unstable and the service quality cannot be satisfied. However, after migrating to ByConity, since the computing group is completely physically isolated, natural resource isolation and tenant isolation can be achieved. The queries of different users are not affected by each other. The overall query performance can reach 91%. All user queries can be completed within 10 seconds. . In addition, ByConity provides self-developed complex query links, self-developed Disk Cache to reduce cold data reading, and indexes frequently used Arrays. Moreover, the hot reading efficiency is also better than that of the original ClickHouse cluster. Compared with the original ClickHouse cluster, ByConity The performance loss on the cluster is within 10%.
-
Low operation and maintenance costs, and faulty nodes can be replaced in seconds :
Originally, in the Clickhouse cluster, if a node in the cluster is found to be broken, the entire machine needs to be taken down for repairs. This is because ClickHouse’s computing resources, storage resources, and metadata information are all on this node, which is equivalent to a small cluster. A computing resource is lost, and a storage copy is lost. Before replacing a new node, the local disk of the broken node needs to be backed up and migrated to the new node. The maintenance cost is relatively high, and data consistency is difficult to guarantee. . For ByConity, if the computing group is broken, since the computing group does not store data and only contains stateless computing nodes, it only needs to be replaced with a new computing group. The reliability and consistency of the data are provided by HDFS. To ensure that the loss of local hot read data cache is controllable on business query performance, this part mainly benefits from the ByConity storage and computing separation architecture.
-
Sensorless expansion and contraction , saving resource costs:
ByConity can achieve seamless expansion and contraction. It is a modular and containerized deployment based on the elastic scalability of Kubernetes. If there are enough machines, it can be expanded indefinitely. At the same time, if the server fails, we don’t have to worry because ByConity's node is only a stateless computing node, and removing it directly will have little impact on the entire cluster. In addition, adaptive scheduling is used to avoid slow nodes, improve throughput capabilities, and improve node resource utilization. At the same time, ByConity’s compression rate is extremely high. Taking one of its businesses as an example, 460TB of data is added every day, reaching 100TB after compression, with a compression ratio of 65%. It also supports low-radix encoding & ZSTD and other compression methods, which consumes more storage in extreme cases. Less than parquet.
-
Data consistency is strongly guaranteed, and maintenance complexity is close to zero :
After migrating to ByConity, we completely solved the data consistency problem, because ByConity does not have local master-slave synchronization problem, and the data consistency problem is directly solved by the underlying object storage, such as HDFS/S3, etc. In this way, the complexity of consistency maintenance is greatly reduced, and the error probability is also lower. Currently, few users report data consistency issues. But it was often encountered before, because the ClickHouse cluster is maintained by multiple copies through inter-node communication, and consistency issues are maintained through consistency queues. The implementation is also very complicated and error-prone. In addition, ByConity can directly access data files through HDFS. Different computing engines adapt to different connectors to read data and have universal capabilities.
future outlook
After a year and a half of practical exploration, ByConity has become the main OLAP engine used internally. A large number of users and data will be moved in later, and it will eventually replace the original ClickHouse cluster. It can be seen that ByConity, as an OLAP engine with separated computing and storage, has the advantages of high performance, high scalability and high stability, and can meet the needs of large-scale data processing and analysis. At the same time, through communication in the community and the Roadmap discussion released by the community https://github.com/ByConity/ByConity/issues/26, ByConity will mainly focus on the following directions in the future:
-
Supports multi-stage execution, ETL capabilities, etc. of the execution layer
-
Support data lake federated queries such as Hudi, Iceberg, etc.
The ByConity community has a large number of users and is a very open community. We invite everyone to discuss co-construction with us on Github. You can also join our WeChat group, Feishu group or Discord to participate in communication.
GitHub：https://github.com/ByConity/ByConity
{{o.name}}
{{m.name}}