On September 26, 2023, Tencent's big data team and the StarRocks community jointly held a grand event called "Building a New Generation of Real-Time Lake Warehouse". The event brought together technical experts from Tencent Big Data, Tencent Video, Tencent Games, Tongcheng Travel and the StarRocks community to discuss in depth the lake-warehouse integrated technology and its application practices and other high-profile topics, with more than 20,000 viewers. .
The future development trends and directions of big data are issues that many developers are concerned about. At the beginning of the event, Chen Peng, head of Tencent's big data research and development, and Zhang Youdong, CTO of Jingzhou Technology, conducted a wonderful technical dialogue from the perspective of industry experts. They expressed that big data will develop in the direction of "one data, all analytics" in the future based on the current hot spots of big data technology, the development of lake warehouse technology, and the development trend of future integrated lake warehouse technology.
As a leading domestic Internet company, Tencent has rich practical experience in integrating lakes and warehouses. Through trial and exploration, Tencent Big Data has expanded and upgraded the Hucang integrated architecture based on StarRocks to provide high-performance, one-stop solutions for the business. At the event, Tencent's big data team shared their advanced experience in integrating lakes and warehouses, including how to build a lake-warehouse integration architecture, the application of lake-warehouse analysis in Tencent's video business scenarios, and how Tencent games have gradually evolved from Lambda architecture to The technical process of the Hucang integrated architecture. Among them, the best practices in storage and calculation separation and hot and cold data stratification have also brought valuable reference to other developments.
At the same time, Mr. Zhou Tao from Tongcheng Travel was invited to introduce how Tongcheng Travel successfully solved the problems in user portraits by introducing StarRocks, improving query efficiency and efficiently implementing complex queries. This article will summarize the important content and video materials of this technical exchange event. At the same time, I sincerely thank every partner in the community for their support and active participation in this event. In the future, we will continue to share more high-quality technical content with you!
Technology Talk: Open Source and Next Generation Lake Warehouse
Chen Peng, Head of Tencent Big Data Industry and Research/Executive Member of Tencent Big Data Technology Committee
Zhang Youdong, CTO of Jingzhou Technology/Member of StarRocks Technical Steering Committee
In this sharing, the two experts deeply discussed the current focus issues of big data technology, the development history of Hucang technology, and the evolution of StarRocks and Tencent in Hucang integration. They also talked about the future trend of integrated lake and warehouse technology.
Chen Peng believes that the development of big data technology should be a gradual refinement process, and the big data system needs to become more refined to make business applications easier. This needs to be achieved through the joint action of data links and big data architecture, rather than just relying on one or two technical points. Therefore, Tencent big data is developing in an integrated direction. This system includes 4 horizontal and 3 vertical dimensions. The four horizontal directions refer to the integration of software and hardware, the integration of resources, the integration of storage and caching, and the integration of computing, which help to build a simpler and more elegant data architecture. The three verticals refer to the comprehensive adaptation and automation of big data through real-time lake warehouses, virtual engines and intelligent platforms.
Zhang Youdong believes that the current amount of data has experienced explosive growth, and the main problem solved by the big data system is how to mine valuable information from massive data. Against this background, in the process of evolving towards the integration of lakes and warehouses, StarRocks has realized that one data supports all analysis scenarios, thus greatly simplifying the data analysis process. This is also consistent with the evolution path of Tencent’s big data.
In general, the future development trend of Hucang will tend to be database-based, simplify processes, and achieve integration, thus promoting the development of intelligent applications.
Technology Talk: Open Source and Next Generation Lake Warehouse
Tencent Tianqiong’s one-stop lake-warehouse integration platform architecture revealed
Chen Jiutian Tencent Big Data Senior Engineer/StarRocks Active Contributor
In this sharing, Jiutian first discussed the problems currently encountered in the industry in the lake-warehouse integration scenario: how to freely circulate lake-warehouse data, how to achieve integrated query of lake-warehouse data, how to optimize the lake-warehouse modeling link, etc., and also introduced Tianqiong. How does the StarRocks Hucang integrated architecture solve the above problems and implement it on a large scale within Tencent’s internal business? This architecture greatly simplifies the user's lake warehouse modeling link while taking into account query performance and storage costs.
Tencent Tianqiong’s one-stop lake-warehouse integration platform architecture revealed
How Tongcheng Travel implements user portrait analysis based on StarRocks
Zhou Tao, Head of Tongcheng Travel Data Center
In 2022, Tongcheng Travel introduced StarRocks to unify OLAP components and has been widely used within the company. At present, it has been successfully used in accommodation, travel and other fields, including BI dashboards, data analysis, indicator systems, risk control, anti-crawling, user marketing and real-time data warehouse and other business fields.
This sharing focuses on StarRocks’ user portraits and CDP (Customer Data Platform) application practices in Tongcheng Travel. Before the introduction of StarRocks, there were problems with user profiling analysis, such as tag importing resources that consumed a lot of resources, import operations affected query performance, only supported wide table queries, and could not handle complex association and aggregation queries. After introducing StarRocks, Tongcheng Travel has optimized the data import function, significantly improved the speed of complex queries, realized efficient association of detailed tables and bitmaps, and better supported key functions such as CDP crowd analysis and export marketing.
How Tongcheng Travel implements user portrait analysis based on StarRocks
StarRocks application practice in Tencent Video
Zhao Xuan Tencent Video Data Engineering Center Big Data Development Senior Engineer
This time we mainly introduce to you the application practice of Tencent Video using StarRocks in Hucang analysis scenarios, as well as the evolution of Tencent Video’s data architecture. By describing the query efficiency, lake warehouse hierarchical model construction and other issues encountered in the lake warehouse analysis scenario, StarRocks' solution for lake warehouse analysis based on Iceberg was shared. In addition, it also introduces the hierarchical model construction method under the StarRocks lake warehouse architecture and the storage method of hot and cold data separation; at the same time, it introduces the use of StarRocks to build indicator services in application practice, through Bitmap, aggregation engine, logical view, indicator acceleration, etc. Facilitate personalized data analysis and build an efficient, easy-to-use, and simple lake warehouse architecture to enhance data value.
StarRocks application practice in Tencent Video
Tencent Games’ integrated exploration of lakes and warehouses based on StarRocks
Huang Yiwen Tencent Game Data Analysis Engine R&D Engineer/StarRocks Active Contributor
This sharing article mainly introduces the evolution technology route of Tencent Games from the original Lambda architecture, the data warehouse architecture based on StarRocks, and the integrated lake and warehouse architecture based on StarRocks. Focusing on the separation of storage and calculation, data hot and cold stratification, and lake-warehouse integrated experience optimization, key construction has been carried out; at the same time, in the implementation stage, in-depth polishing has been carried out on asynchronous materialized views, query performance optimization, and offline import performance, so as to achieve both Easy-to-use Hucang integrated architecture for performance and cost.
Tencent Games’ integrated exploration of lakes and warehouses based on StarRocks
This article is published by mdnice Multiple platforms