For more technical exchanges and job opportunities, please follow the WeChat official account of ByteDance Data Platform and reply [1] to enter the official communication group
As the next iteration of cloud computing, serverless allows developers to focus more on building the applications in the product without having to consider the underlying stack issues. As the maturity of related technologies has increased in recent years, the market's acceptance of Serverless has also become higher and higher. It can be said that today, Serverless has entered a high-speed track of development toward maturity and stability.
As a cloud-native data warehouse launched by Volcano Engine, ByteHouse is built based on the open source ClickHouse, and has further upgraded the OLAP engine capabilities, performance, operation and maintenance, and architecture under the test of ByteDance's internal and external scenarios. In addition, ByteHouse is also exploring the direction of Serverless, building a new generation of data warehouse based on the concept of cloud-native, with three layers of decoupling in the architecture. It is expected that with the support of Serverless, it can provide more stable, reliable, and Trusted analysis services free developers' time and energy from infrastructure operation and maintenance optimization, and focus more on core business functions.
However, there are still some technical difficulties in the implementation of Serverless technology in the OLAP field.
First of all, OLAP data analysis involves almost full-stack technical points in IT fields such as storage, network, operating system, database, and AI, requiring manufacturers to make sustained and high-cost R&D investments. Moreover, it is difficult for these investments to see market returns in the short term. Once they stop midway, it means that all the initial investment will be "wasted".
Secondly, although there are several commercial serverless architecture data warehouses, the computing power scale they provide is difficult to support the needs of medium and large-scale data warehouses or analysis platforms. In other words, the serverless architecture data warehouse still needs to be further improved in terms of computing power.
Finally, the large-scale application of Serverless also faces the problem of service standardization. Without standardized specifications, users will be locked into the platform, making it impossible to achieve translational and seamless relocation of applications. Therefore, in order to promote the large-scale application of Serverless, a supporting standard and specification system is needed.
In order to better solve the above difficulties, Volcano Engine ByteHouse gradually evolved to a Serverless architecture and decoupled the three-layer architecture for the storage layer, middle layer and cloud service layer.
The technical architecture of ByteHouse’s evolution to Serverless
In the design of the storage layer, ByteHouse achieves serverless, elastic scaling, and unlimited capacity expansion. In order to improve the performance issues under the separation of storage and computing architecture, ByteHouse optimizes HDFS semantics to reduce latency by 3 times with only a 10% increase in bandwidth. It also uses connection multiplexing, RDMA, transmission compression and other technologies for network communication. Greatly alleviates the network amplification problem.
In the middle layer, ByteHouse achieves load isolation by isolating by read and write, isolating by application category, etc., and while meeting the basic needs of users, the architecture gradually evolves to serverless.
In the upper cloud services layer, ByteHouse provides centralized catalog metadata services, cluster management services, etc., decoupling metadata from the computing layer, achieving statelessness, and obtaining second-level elastic scaling and start-stop capabilities.
The application of Serverless in the OLAP field still faces some challenges, but with the continuous evolution and iteration of technology and the establishment and improvement of standardized specification systems, ByteHouse will continue to explore and apply the advantages of Serverless architecture to provide better data analysis for the industry and users. Serve.
Click to jump to ByteHouse to learn more
OpenAI opens ChatGPT to all users for free. Voice programmers tampered with ETC balances and embezzled more than 2.6 million yuan a year. Spring Boot 3.2.0 was officially released. Google employees criticized the big boss after leaving the company. He was deeply involved in the Flutter project and formulated HTML-related standards. Microsoft Copilot Web AI will be Officially launched on December 1st, supporting Chinese Microsoft's open source Terminal Chat Rust Web framework Rocket releases v0.5: supports asynchronous, SSE, WebSockets, etc. The father of Redis implements the Telegram Bot framework using pure C language code . If you are an open source project maintainer, encounter How far can you endure this kind of response? PHP 8.3 GA