On October 21, the Doris Summit Asia 2023, the first offline technology summit of the Apache Doris community, hosted by Feilun Technology and co-sponsored by Alibaba Cloud and Tencent Cloud, came to an end with great enthusiasm.
With the theme of "Walking with Innovators", this summit has a main forum and four parallel forums: smart finance and government and enterprises, advanced intelligent manufacturing and telecommunications, corporate services and new economy, and Internet and entertainment. Participants from Meituan, UnionPay Commerce, Postal Savings Bank of China, Alibaba Cloud, Tencent, Xiaomi, Huawei, NetEase, Ping An, Zhongan Insurance, Wuxi Xishang Bank, China Unicom, China Telecom, Emma, Yurun, Qi'anxin, Huya, Lalamove, Quwan Dozens of technical experts from industry innovation leaders such as technology, 360, Vast Depth, Horizon, Observation Cloud, Chinalco Vision, etc. attended the summit and contributed wonderful technical speeches, sharing industry best practices and multi-scenario solutions based on Apache Doris program to jointly discuss the most cutting-edge technologies and future trends in the field of data analysis. This summit also attracted the attention of a large number of technology enthusiasts. Nearly a thousand big data and data field practitioners came to the site to communicate face-to-face with technical experts. More than 100,000 people watched the live broadcast of the summit online, providing a great opportunity for the majority of data technology practitioners. Provides an annual technology feast.
Join the Innovators, Apache Doris in 2023
In the opening speech of the summit, Yi Guolei, a member of Apache Doris PMC and vice president of technology of Feilun Technology, reviewed the important version iterations and community development summary of Apache Doris in the past year, focusing on the theme of "Walking with Innovators", and shared the response Rethinking the real-time analysis of massive data, and releasing the future evolution direction and functional planning of Apache Doris.
Looking back on the past few years, Apache Doris has been accelerating its evolution, especially in 2023, Apache Doris released the landmark version 2.0, officially entering the 2.0 era . A series of notable achievements have been achieved in this version:
-
Introducing an adaptive parallel execution model and a new query optimizer, blind test performance is improved by 10 times, multi-table association is improved by 13 times, single table scenarios are improved by 10 times, and high concurrency point queries are improved by 20 times;
-
Expand from typical OLAP scenarios such as reports and Ad-hoc to lake-warehouse integration, high-concurrency data services, and log retrieval and analysis to support more unified and diverse analysis scenarios;
-
Supports high-throughput writing of real-time data, second-level latency, complete support for all types of data updates, and builds a more efficient, easy-to-use and stable real-time data processing and analysis link;
Behind the performance leap and scene expansion, it is inseparable from the hard work of all community developers. Up to now, the community has nearly 600 contributors, with an average of more than 120 monthly active contributors. It has become one of the most active open source projects in the world of big data and databases . The source of contributors is also more diversified, with more than 100 benchmark companies widely distributed in dozens of industries at home and abroad. It has also attracted investment from many top domestic cloud vendors, including Alibaba Cloud, Tencent Cloud, Baidu Smart Cloud, Huawei Cloud, Tianyi Cloud, Volcano Engine, etc. At the same time, related products cover almost all mainstream cloud platforms at home and abroad.
At the same time, this year, Apache Doris gained recognition from more industry users. At the summit, Yiguolei announced for the first time that Apache Doris has more than 4,000 enterprise users worldwide , covering dozens of industries such as finance, Internet, energy, manufacturing, communications, and enterprise services. It is at the core of many large enterprises. It is widely used in analysis business. With such a large user scale, Apache Doris has become the first choice for users from all walks of life to use real-time data warehouses, and has become the de facto standard in the field of open source real-time data warehouses!
These important developments not only demonstrate Apache Doris's continuous innovation capabilities and technical strength, but also prove our in-depth understanding and accurate grasp of user needs. Yi Guolei said that in the future, Apache Doris will continue to be committed to promoting innovation in real-time data analysis technology, including: Real-time analysis: achieving ultimate query performance on large-scale real-time data;
-
Integration and unification: Provide support for multiple analysis workloads in one system, simplifying the operation and maintenance costs caused by complex architectures;
-
Cloud-native: Innovate cloud computing infrastructure and use the extreme elasticity of the cloud to reduce storage and computing costs;
Drive business innovation and development with technological innovation
At the main venue of the summit, Zeng Linxi, senior technical expert and query engine team leader of Meituan, shared the practice of Apache Doris in Meituan's unified OLAP engine.
As a technology retail company, Meituan continues to promote the digital upgrade of service retail and merchandise retail on the demand and supply sides to provide consumers with quality services. Data is the foundation of refined operations. Behind Meituan's rich business scenarios, it faces the demand for efficient and real-time analysis of massive data. In order to meet this challenge, Meituan has gone through the selection and evolution of multiple OLAP engines and explored the construction plan of a unified OLAP engine. After comprehensively considering various factors such as technology stack matching, code scalability, project implementation cycle, and business migration costs, Meituan finally chose to build a unified OLAP engine based on Apache Doris.
So far, Meituan has launched nearly 200 clusters, nearly 10,000 physical nodes, and more than 100 million query and analysis requests every day . While serving the business, Meituan is also deeply involved in the Apache Doris open source community. It has in-depth co-construction with the community in important research and development directions such as query optimizer, vectorized execution engine, pipeline execution framework and regression testing framework. It is the most powerful contribution to the community. one.
At the main venue of the summit, Yang Jinxiong, senior data architect of UnionPay Business, shared his practical experience in building a new generation of real-time data warehouse based on Apache Doris. UnionPay Business is a large non-bank payment institution in China, providing comprehensive payment services based on bank card acquiring and online payment, as well as diversified and professional merchant value-added and technological innovation services. It has served more than 25 million merchants, installed more than 39 million terminals, and has ranked first among Asia-Pacific acquirers in terms of number of acquiring transactions for ten consecutive years. Faced with such large-scale business growth, traditional offline data warehouse platforms can no longer efficiently support business needs.
"Apache Doris can import and synchronize data quickly and accurately, while obtaining faster response times during query analysis. After the introduction of Apache Doris, import performance is improved by 2-5 times, ETL scenario performance is improved by 3-12 times, and query analysis response The speed is increased by 10-15 times , and the significant performance improvement provides strong support for our business growth and decision-making."
By introducing Apache Doris, UnionPay Business has upgraded its real-time data warehouse, solved the problem of rapid business and data growth, and met the financial industry's needs for large-scale data processing and real-time queries. The successful practice of UnionPay Business provides financial companies with reference value, demonstrating the huge potential of Apache Doris in promoting financial technology innovation.
In an era when mobile payments are popular, financial anti-fraud has become an important means to protect the security of corporate and customer assets. Di Xiaoran, a technical expert from the Postal Savings Bank of China Software R&D Center (Xi'an Branch), introduced at the main forum that the Postal Savings Bank of China has built an intelligent risk control and anti-fraud platform based on Apache Doris. The platform has monitoring and alarming, ad hoc analysis, and correlation analysis. and rapid positioning and other functions to meet the needs of all-round risk prevention and control.
"Time is a key metric in financial anti-fraud operations, so rapid operational analysis and quick acquisition of key value clues are crucial. The anti-fraud platform is based on the powerful functions of Apache Doris, bringing a series of important performance improvements and application value: The response speed of core reports has been shortened from days to seconds, 80% of ad hoc analysis can achieve second-level response, alarm efficiency is as high as 95%, alarm delay does not exceed 5 seconds, and storage costs are reduced by 70%.... ”
The Postal Savings Bank of China's intelligent risk control and anti-fraud platform uses Apache Doris to achieve efficient and real-time data support, allowing the Postal Savings Bank to better respond to financial fraud and protect the company's own interests and the security of customer assets.
Growing towards the cloud, the choice of more industrial innovators
With the advent of the cloud era, enterprises have an increasing demand for flexible and efficient cloud computing. Whether it is a public cloud, private cloud or K8S container platform, more and more enterprises hope that Apache Doris can provide deeper adaptation to the new infrastructure of cloud computing in order to provide more flexible and powerful elastic capabilities.
Ma Ruyue, CEO of Feilun Technology and member of Apache Doris PMC, introduced SelectDB, a modern real-time data warehouse built on Apache Doris. Currently, SelectDB has two product forms: SelectDB Cloud, a cloud-native data warehouse service based on multi-cloud, and SelectDB Enterprise, an enterprise version that meets privatized deployment requirements , which can meet the needs of different users.
Among them, SelectDB Cloud achieves complete separation of storage and computing, and provides functions such as multi-computing cluster load isolation and elastic computing expansion and contraction. This version will also be contributed to the Apache Doris community in the future to help enterprise users achieve more efficient data analysis and processing. , more flexible elastic capabilities and lower usage costs.
In March this year, Alibaba Cloud and Feilun Technology formally reached a strategic cooperation agreement. The two parties decided to jointly develop a new generation of real-time data warehouse called "Alibaba Cloud Database SelectDB Edition" to provide users with fully managed services on Alibaba Cloud. This innovative cooperation aims to combine modern real-time data warehouse technology with Alibaba Cloud's powerful infrastructure capabilities to bring users a more excellent data analysis and processing experience.
At the summit, Alibaba Cloud and Feilun Technology jointly released the Alibaba Cloud Database SelectDB version. This product is deeply optimized based on Alibaba Cloud IaaS and can be seamlessly integrated with other Alibaba Cloud products, giving users a better cloud experience and truly explaining the " Doris on the cloud, more than that" concept .
Peng Xiang, a senior technical expert on Alibaba Cloud database, said that the powerful alliance between Alibaba Cloud and Feilun Technology demonstrates the determination and innovation capabilities of both parties in promoting the development of database technology and improving user experience.
Li De, a member of Apache Doris PMC and head of Tencent Cloud TCHouse-D R&D, shared that there is a significant difference between enterprise-level needs and open source projects. Enterprise users are more pursuing stable availability, compatibility and adaptation of upstream and downstream ecosystems, timely technical support responses, and complete product information. Compared with open source users, enterprise users have more comprehensive and strict needs.
In order to meet the needs of enterprise users, Tencent Cloud built the TCHouse-D database based on the Apache Doris kernel, which can provide enterprise users with easy-to-use, flexible and stable real-time data warehouse services . Li De introduced in detail the features and advantages of TCHouse-D, which can be set up in just a few minutes and can easily handle real-time query and analysis of massive data.
At the same time, Tencent Cloud will also accumulate problems and experiences after launching enterprise-level products, continue to feed back to the community, and will continue to invest in community building in the future.
More practical applications from industry innovators
In addition to the main venue, dozens of technical experts from four parallel forums: Smart Finance and Government and Enterprises, Advanced Intelligent Manufacturing and Telecommunications, Enterprise Services and New Economy, and Internet and Entertainment also brought wonderful insights to the participants at each venue. Speeches shared the latest technology trends and practical experiences in their respective fields.
-
At the Smart Finance and Government-Enterprise Forum , many technical experts from Ping An Life, Zhongan Insurance, Wuxi Xishang Bank, Financial OneConnect, Huawei, Zhida Xinke, as well as innovations from Shushi Technology, Tapdata, Feilun Technology, etc. Representatives from pioneering companies shared with all participants how to deal with the challenges of real-time analysis and provided an in-depth analysis of Apache Doris' best practices in industry scenarios. At the same time, Apache Doris PMC member Chen Mingyu also revealed the technical implementation of Apache Doris in the Hucang federation analysis scenario in this forum. Apache Doris PMC members Li Haopeng and Feilun Technology Xiong Zhongjian shared the thinking and optimization practices behind the 10-fold improvement in blind test performance of Apache Doris in version 2.0.0.
-
In the Advanced Intelligent Manufacturing and Telecommunications Forum , technical experts from industry pioneers such as China Telecom, China Unicom Western Innovation Research Institute, Haohan Depth, Tianyi Cloud, Emma Electric Vehicle, Horizon, Chinalco Vision, Xiaomi Group and Feilun Technology, Real-life production practice cases of Apache Doris in telecommunications and manufacturing scenarios were shared with participants, and they discussed better solutions for real-time analysis of massive data. Wang Huajie, senior architect of Feilun Technology and member of Apache StreamPark PPMC, shared the best practice of synchronizing the entire database of multiple data sources to Apache Doris.
-
In the Internet and Entertainment Forum , big data experts from industry pioneers such as NetEase Games, Quwan Technology, Tencent, Huya, Lalamove, 360, and Alibaba Cloud shared with participants how Apache Doris helps companies achieve success in different business scenarios. Precision marketing and business growth. At the same time, Wang Tianyi, a senior solution architect from Feilun Technology, will also share with you a practical solution for building a user behavior analysis and profiling platform based on Apache Doris.
-
In the Enterprise Service and New Economy Forum , many technical experts from Observation Cloud, Qi’anxin, Yurun, Wanwu Cloud City, Tencent Cloud, Jiuzhang Arithmetic, Feilun Technology and the community shared how to provide customers in different industries with Apache Doris. Services to meet large-scale, personalized data processing and analysis needs, while sharing scenario solutions such as data integration and log analysis.
At this point, Doris Summit Asia 2023 has come to a successful conclusion. I would like to thank all the speakers, partners, all online and offline participants who participated in this conference, all technical practitioners who support and recognize Apache Doris, and open source technology enthusiasts.
Here we also pay tribute to every innovator who is chasing the wind and the moon, and we look forward to working with more innovators to explore more possibilities in the data world!
How to get summit replays & materials?
We are urgently sorting out the full replay of the summit and speech PPT, and expect to make them public in 1-2 weeks. Please continue to pay attention to the "SelectDB" official account and the open source technology forum ( http://forum.selectdb.com/ ) to receive relevant notifications and push notifications as soon as possible.
Alibaba Cloud suffered a serious failure and all products were affected (restored). Tumblr cooled down the Russian operating system Aurora OS 5.0. New UI unveiled Delphi 12 & C++ Builder 12, RAD Studio 12. Many Internet companies urgently recruit Hongmeng programmers. UNIX time is about to enter the 1.7 billion era (already entered). Meituan recruits troops and plans to develop the Hongmeng system App. Amazon develops a Linux-based operating system to get rid of Android's dependence on .NET 8 on Linux. The independent size is reduced by 50%. FFmpeg 6.1 "Heaviside" is released