Big data is one of the main means to support business operations and performance growth in the digital transformation of enterprises. Real-time and cloud-native have become an inevitable trend in the development of big data technology.
On April 18th, Volcanic Engine Spring FORCE Power Conference was held in Shanghai. At the meeting, Volcano Engine released a cloud-native big data real-time computing platform product - streaming computing Flink version. Born out of ByteDance's largest real-time computing cluster practice in the industry, streaming computing Flink products have been explored in scenarios such as real-time ETL, real-time data warehouse/lake, real-time machine learning, real-time risk control, etc., to help customers build new growth drivers on the cloud and facilitate business agile innovation.
Byte practice - daily peak 10 billion QPS
Beginning in 2017, ByteDance began to try to use Flink as the main streaming computing engine. In the following two years, the streaming computing team supported businesses such as real-time sample splicing, model training, and real-time recommendation algorithms within Byte, and completed 100% migration of JStorm jobs within the company. By 2019, the application of Flink within Byte has expanded rapidly, covering almost every product including Douyin, Toutiao, and Xigua. At the same time, the team began to actively participate in the joint construction of the community. In 2020, Li Benchao was invited to become the Apache Flink Committer. In the past two years, the team has also conducted a lot of exploration in the direction of Flink OLAP. Scheduling, runtime, SQL and other aspects have been fully optimized to greatly improve performance. A single cluster can support 200+ QPS. It has been implemented in more than ten business scenarios such as User Growth, Feishu, and e-commerce. The daily query scale exceeds 50W times.
So far, the real-time business scenarios built based on streaming computing Flink have involved almost all businesses and products of Byte, including real-time data warehouses, real-time risk control, commercialization, e-commerce, games, novels, education, real estate, finance, etc., with daily real-time peaks
exceeding
10 billion QPS . At the same time, stream-batch integration has been widely used and implemented in video cloud, real-time counting features, e-commerce, SQL data synchronization and other scenarios, and 6K+ Flink Batch SQL tasks have been launched.
Enterprise-level Enhancement - Streaming Computing Flink Edition
The Flink version of the volcano engine streaming computing relies on ByteDance's practice in the industry's largest real-time computing cluster.
The Volcano Engine streaming computing Flink version is based on the Volcano Engine container service (VKE/VCI), providing serverless extreme flexibility, and is a new generation
of cloud-native
fully managed
real-time computing platform
out of the box . On the premise of being 100% compatible with the open source Flink, deeply optimized 30+ enterprise-level product enhancements, including the following features:
-
Improve development efficiency. The streaming computing Flink version supports operator-level Debug output, Queryable State, and Temporal Table Function DDL, which has significantly improved development efficiency compared to the open source version of Flink.
-
Improved reliability. The Flink version of streaming computing performs Checkpoint for a single Task, which improves the success rate of Checkpoint under large concurrency. Single-point task recovery and node blacklist mechanism functions ensure rapid response to failed nodes and avoid overall business restart.
-
Serverless cloud native architecture. Extreme flexibility, 1‰ core fine scheduling.
-
Enhanced usability. Simplified SQL development, out-of-the-box, O&M-free, supports full lifecycle management of streaming data.
-
High performance at low price. High cost performance, high SLA guarantee, ultra-low TCO.
The overall structure of the volcano engine streaming computing Flink product
In terms of the overall architecture, the Source/Sink side supports multiple types of data storage, and builds an extremely flexible and flexible resource scheduling platform with the help of container cluster infrastructure; the engine layer achieves the unification of Runtime & API in terms of streaming and batch integration, and builds high-level auxiliary development capabilities such as intelligent diagnosis and automatic tuning through the service platform.
At present, the Flink version of the volcano engine stream computing product has formed a product solution that integrates multiple capabilities such as computing, storage, and intelligence. While having cross-cloud and multi-cloud capabilities, it will always adhere to the development direction of
big data
technology "cloud-native, real-time, and intelligent"
to provide the impetus for the digital transformation of enterprises.
{{o.name}}
{{m.name}}