Introduction to this article : With the continuous expansion of the scale of online live broadcasts, online knowledge services have ushered in broad development opportunities in the live broadcast industry. As a technical service provider with user service as its core, Xiaoetong attracts and converts potential users through multi-platform live broadcast and push streaming services. At the same time, through the live broadcast data analysis platform, it helps merchants analyze live broadcast indicator data, analyze user viewing conditions, and build accurate User portrait. However, with the surge in the number of live broadcast users, the platform's support for writing and querying gradually failed to meet business needs. In order to cope with the challenge, Xiaoetong embarked on an architecture upgrade journey. This article will introduce in detail the optimization process of Xiaoetong's live broadcast data analysis platform, share Xiaoetong's practical experience in optimizing writing and query performance, improving user tag functions and ensuring platform stability based on Apache Doris, providing merchants with more refined User operation support . (Author | Gong He, Xiaoetong big data development engineer)

Shenzhen Xiaoe Network Technology Co., Ltd. is a technical service provider with knowledge products and user services as its core. Since its founding, it has produced a total of 20 million knowledge products and served 1.6 million merchants, including Tencent Academy, Huaxia Fund, Xibei, Wu Xiaobo Channel, Ten O'clock Reading, Caesar Travel and other first-line well-known brands in various industries. As a one-stop tool for private domain operations, Xiaoetong provides multi-faceted closed-loop solutions for merchants' business pain points, including product and service delivery, marketing customer acquisition, user operations, organizational role management, brand value output, etc. to achieve Business upgrade.

With the rise of the "live streaming to bring goods" consumption model, the scale of online live broadcasts continues to rise, and online knowledge services have begun to open up vast areas of development in the live broadcast industry. The power of live broadcast tools can effectively attract and convert potential users, and the live broadcast analysis platform can Realize the retention and analysis of C-end user data and assist B-end merchant customers in refined user operations.

In this context, Xiaoetong Live Broadcast provides merchants with multi-platform live broadcast and streaming services, as well as a variety of live broadcast marketing methods to bring goods. It can not only guide public and private domain traffic to the live broadcast room to achieve merchants’ traffic diversion goals, but also Provides a multi-dimensional analysis platform. Merchants can export and analyze various live broadcast data indicators through the live broadcast data analysis platform and generate user tags to understand the situation and dynamics of users watching live broadcasts and build accurate user portraits.

At present, the number of Xiaoetong live broadcasts has accumulated to 6 million, with nearly 800 million end users, and the highest number of simultaneous online viewers exceeds 10 million. However, in the face of the surge in the number of live broadcast users, the live broadcast data analysis platform's support in data writing and checking is gradually unable to meet business needs. We hope to use the analysis platform to grasp the number of people online in the live broadcast room, online time, product conversion rate, transaction volume and other data , and record the traffic trend chart of the above data during the peak and trough periods of the live broadcast room. In order to meet the above business needs, Xiaoetong puts forward higher performance requirements for the platform architecture:

Writing performance : It is necessary to have the ability to write massive data in real time to ensure stable data import in high-throughput scenarios during live broadcasts;
Data update : In order to cope with changes in transaction status and user data during the live broadcast, we need to structure an efficient and accurate update mechanism to ensure that business personnel have the latest data in a timely manner;
Query performance : It can support high-concurrency enumeration, and can also meet the needs of federated queries, ensuring the timeliness of data analysis in different query scenarios;
Platform operation and maintenance : Since the live broadcast business has obvious peak and trough periods, the analysis platform needs to ensure stable operation during the peak hours of the live broadcast. At the same time, during trough periods, the platform should support elastic node scaling to improve resource utilization and control operation and maintenance costs. In addition, in order to avoid site damage caused by system operation failures and to maintain high system availability, the architecture also needs to provide reliable disaster recovery solutions to ensure business continuity.

After encountering system business bottlenecks, Xiaoetong started a journey of platform infrastructure reconstruction. After business research, selection comparison, and internal testing of applications, it iteratively upgraded the original Elasticsearch and HBase architectures based on Apache Doris, and In May this year, the Apache Doris version of the live broadcast data analysis platform was officially launched.

This article will introduce in detail the optimization process of the Xiaoetong live broadcast data analysis platform, share practical experience on how Xiaoetong improves platform query performance, improves user tag functions, and ensures platform stability based on Apache Doris, and ultimately empowers merchants to refine user management.

Based on early architecture of Elasticsearch and HBase

The original data of the live broadcast data analysis platform mainly comes from MySQL's business database, learning SDK, and buried point SDK. The platform architecture is mainly divided into offline and real-time links:

Offline link: User-related data is collected through DataX and stored in HDFS in full. It is calculated and written into HBase by Hive to form a master table of basic user information.
Real-time link: Relevant indicator data (such as real-time subscription, access, live interaction, user rights, etc.) in the MySQL business library and learning SDK are collected through Flink CDC and reported to Kafka. Flink is used to calculate and store it in Redis in real time.

Based on business needs, the two types of data associate basic user information with live broadcast-related business indicators to form a large table of live broadcast participating users. When the main business data such as rights and subscriptions change, they need to be associated and supplemented with the user dimension table in HBase. Full data. Every time the data is updated, after the wide table is completed, all fields will be sent to downstream Elasticsearch to carry the query needs of business personnel and avoid concurrent modifications.

As the scale of live broadcast users expands, our requirements for data timeliness and query performance have gradually increased, and the limitations of the original architecture have gradually affected business analysis needs:

The timeliness of data flow is low: Due to too many components, the computing link is too long and there are too many nodes. The timeliness of writing and querying will inevitably be affected, and it cannot support real-time import and extremely fast analysis during live broadcast peak periods.
The data update link is complex: every time the live broadcast data changes, the changed data needs to be written into the main table and dimension table first, and then re-associated to generate a new wide table to overwrite the old data. This process involves multiple components and long links, increasing the complexity and inefficiency of the update mechanism.
Insufficient query performance: In high-concurrency query scenarios, Elasticsearch needs to handle a large number of query requests and data access, resulting in heavy query load and low analysis efficiency; in associated query scenarios, Elasticsearch cannot meet the Join calculation requirements, and query flexibility is low.
High architecture operation and maintenance costs: Too many components will also lead to higher overall architecture operation and maintenance costs, increased developer learning costs, and limited indicator expansion.

Based on the above architectural pain points, we started the research work on the system reconstruction project in July 2022, and finally selected Apache Doris among many data warehouses for architecture upgrade, and started a 6-month internal testing work in November of the same year. We hope to leverage the advantages of Apache Doris in real-time writing, efficient data updating, extremely fast query performance, and simple operation and maintenance to strengthen live user data mining in live data analysis scenarios and obtain more high-value business insights.

A new live data analysis platform based on Apache Doris

After upgrading to a new architecture, we unified the two data processing links, real-time and offline, and handed them all over to Apache Doris for processing. As shown in the figure, after the original data is calculated by Flink stream processing, it will be written into the table model of Apache Doris in real time. According to the scenario, we have selected different data models for query acceleration. The data table and corresponding query requirements are as follows:

User dimension table: Use the Aggregate Key model of Apache Doris to aggregate user dimension data, and update some columns through replace_if_not_null;
Tag table: In order to better build the user tag system and update user portraits in real time, we use Unique Key to divide the data into tag and tag version tables, update the data through Merge on Write, and use Doris related query performance to enable merchants to obtain Latest version of user tag.
Learning data table: The Aggregate Key model is also used to aggregate users’ learning data in the platform to support subsequent live broadcast user analysis needs.

After the introduction of Apache Doris, it provided us with a new data flow to meet the needs of writing and query timeliness, and effectively solved the pain points of the original architecture link. Currently, we provide real-time writing and extremely fast analysis for the live broadcast data analysis platform based on the scale of more than ten Doris clusters, and have brought the following specific benefits:

01 The writing and checking interface is unified and the writing efficiency is higher

Flink-Doris-Connector has built-in Flink CDC, which supports full and incremental data extraction from business databases. At the same time, Doris can also subscribe to data in Kafka through Routine Load and perform first-class multi-table second-level writing, which greatly improves data integration and synchronization. s efficiency.

Use Apache Doris to replace multiple components such as HBase and Elasticsearch in the original architecture, reducing the flow of intermediate data in different links, solving the problems of too long links and too many components, and achieving unified external data through a system of Apache Doris Serve.

02 Data is merged on write and updated efficiently

After Apache Doris aggregation, when the maximum size of a single table reaches billions, it can support 200,000 data Upsert and Insert Overwrite per second, meeting the high throughput writing requirements during peak live broadcast periods.

When data is updated, there is no need to complete the fields and then re-develop the wide table. You can directly use the Merge On Write function to perform lightweight Merge. By merging on write, you can achieve micro-batch high-frequency real-time writing and partial column updates.

03 Query acceleration in multiple scenarios

During high-concurrency enumeration, we frequently use the Doris partitioning and bucketing pruning function to filter unnecessary data when querying, allowing data scanning to be quickly positioned and accelerating query response time.

During related queries, Apache Doris supports Joining multiple 1 billion-level tables to meet analysts' needs for related queries between basic user information and learning, label data, etc. More importantly, Doris can achieve query responses in seconds or even milliseconds in tag retrieval, fuzzy query, sorting, paging and other scenarios.

04 Shorten the development cycle and reduce operation and maintenance costs

Apache Doris has a simple architecture. The FE and BE nodes can be flexibly expanded and reduced, making it easy to deploy, migrate, and operate. After the architecture upgrade, our operation and maintenance costs were reduced by nearly 2/3 compared to before by simplifying links.

At the same time, Doris is compatible with MySQL, making the development operation simple, the threshold for use low, and the research and development costs also reduced. Now the indicator development time has been greatly shortened from one month to a single indicator to only one week, accelerating the indicator iteration cycle.

Next, we will share the application practices of Apache Doris in live data analysis scenarios from three aspects: query acceleration, disaster recovery and backup, and data visualization analysis. We hope to provide readers with more insights into Doris applications through sharing.

User portrait scenario query acceleration practice

For the live broadcast business, user tags are the core factor that constitutes user portraits. It can analyze and refine the behavioral data generated by users during the live broadcast process and generate tag words with differentiated characteristics to better understand user attributes, user preferences, and live broadcast. Habits and other information. In the Xiaoetong live broadcast analysis platform, merchants can define their own rules to generate labels. For example, users who "watched the live broadcast for more than one minute" are one category of labels, and users who "visited the live broadcast room in the last minute" are another category of labels. In this process, merchants will frequently "label" (insert data) or "tear labels" (delete data) on users. At the same time, each user may also have multiple automatic or manual labels. Based on this, the data warehouse needs to have the following capabilities:

Faced with frequent label changes, the data warehouse needs to support real-time writing and updating, and ensure cluster stability.
Facing multiple categories of tags from the same user, the data warehouse also needs to provide powerful associated query capabilities and support fast query and retrieval performance.

We take the following SQL as an example to introduce how to quickly extract data in tag scenarios based on Apache Doris, how to create tag tables and tag version tables through the primary key model, and how to enable fuzzy query parsing for strings to achieve rapid retrieval of user information and accelerate user Behavioral and operational business analysis.

Create a tag table:

create table db.tags (  
u_id string,  
version string,  
tags string
) with (  
'connector' = 'doris',  
'fenodes' = '',  
'table.identifier' = 'tags',  
'username' = '',  
'password' = '',  
'sink.properties.format' = 'json',  
'sink.properties.strip_outer_array' = 'true',  
'sink.properties.fuzzy_parse' = 'true',  
'sink.properties.columns' = 'id,u_id,version,a_tags,m_tags,a_tags=bitmap_from_string(a_tags),m_tags=bitmap_from_string(m_tags)',  
'sink.batch.interval' = '10s',  
'sink.batch.size' = '100000' 
);

Create a label version table:

create table db.tags_version (  
id string,  
u_id string,  
version string  
) with (  
'connector' = 'doris',  
'fenodes' = '',  
'table.identifier' = 'db.tags_version',  
'username' = '',  
'password' = '',  
'sink.properties.format' = 'json',  
'sink.properties.strip_outer_array' = 'true',  
'sink.properties.fuzzy_parse' = 'true',  
'sink.properties.columns' = 'id,u_id,version',  
'sink.batch.interval' = '10s',  
'sink.batch.size' = '100000'  
);

Two types of table writing:

insert into db.tags
select
u_id,  
last_timestamp as version,
tags
from db.source;  
  
insert into rtime_db.tags_version
select 
u_id,  
last_timestamp as version
from db.source;

Bitmap index speeds up tag queries:

After the writing is completed, the business personnel begin to query and analyze the user-related data. Taking a common query scenario as an example, we need to find the label data of a user with the last name Zhang in the latest version. First, we will use LIKE to match the user name that meets the requirements in the user main table. Secondly, we will create a Bitmap for the label field in the paging query. Index, such as "123", "124", "125", "126", "333", and finally find the tag information quickly through three-table association.

with t_user as (
    select 
           u_id,
           name
    from db.user
    where partition_id = 1
    and name like '%张%'
),

 t_tags as (
         select 
                 u_id, 
                 version
         from db.tags
         where (
                   bitmap_and_count(a_tags, bitmap_from_string("123,124,125,126,333")) > 0 
           )
 ),
 
 t_tag_version as (
         select id, u_id, version
         from db.tags_version
 )

select 
    t1.u_id
    t1.name
from t_user t1
join t_tags t2 on t1.u_id = t2.u_id
join t_tag_version t3 on t2.u_id = t3.u_id and t2.version = t3.version
order by t1.u_id desc
limit 1,10;

In a scenario where there are hundreds of millions of live broadcast users and each user has thousands of tags, after using the Bitmap index to query user-related data, the query time can respond within 1 second and accurately provide the business personnel with the required results, which greatly The speed of tag query is accelerated.

Dual cluster high availability solution

During the live broadcast course promotion period, in order to promote the ordering atmosphere in the live broadcast room and increase users' enthusiasm and order rate, the platform will launch lottery and other activities in the live broadcast room. Before introducing Doris, we backed up data daily to object storage. When there is a failure in coupon collection, order placement, etc. during the live broadcast, it takes several hours to go from user feedback, reporting to each level, and then to failure recovery.

However, during these hours, hundreds of thousands of users will be lost, and we need to bear the huge risks and losses of users quitting the live broadcast room and looking for other alternative products. Therefore, when the platform is faced with the impact of massive user access, the speed of data recovery and fault resolution has become the key to the business benefits of the live broadcast room.

In order to ensure the stable operation of the platform during short-term exponential business peaks, we decided to adopt a new disaster recovery plan. Based on the Apache Doris master-standby dual cluster method, hot backup is implemented; Airflow is used to back up daily data to object storage through the Backup and Freeze strategies to achieve cold backup. The specific process of hot backup is as follows:

Double data writing : Modify the Flink Doris Connector and write data to two clusters or a single cluster through configuration. When dealing with a scenario where a certain cluster needs to be upgraded, first stop writing to one cluster and then modify the other cluster, effectively ensuring the cluster's stability. stability.
Load balancing : Switch query requests to specified clusters through reverse proxy. When a problem occurs in a certain cluster, we can promptly send the request to another cluster.
Monitoring and verification : Regularly verify the consistency of the data and table structure of the master-slave cluster to detect data inconsistencies in a timely manner.

We will also further introduce the CCR cross-cluster data replication function launched in Doris 2.0 version, trying to use this function to reduce development and operation costs, and reduce the risk of DDL inconsistency between the two clusters caused by manual reserve cluster double-writing.

Self-developed data visualization platform, unified real-time query export

The data visualization analysis platform is mainly used in three major scenarios: offline reporting, ad hoc query, and real-time data large screen (Dashboard) to provide visual analysis functions for business personnel, management, etc. After introducing Doris, we unified various data sources such as MySQL, Elasticseach, and local files to Apache Doris for implementation .

At the same time, in order to better present data to merchants, we have also added data source management and data set creation functions. Business personnel can directly select data sources in the platform for configuration, testing and other operations to form a visual list. In addition, the platform combines rich chart styles in BI tools to meet the personalized needs of merchants in configuring data dashboards.

After the introduction of Apache Doris, functions such as data management and permission management were realized through the configuration of data sets, and Apache Doris' extremely fast multi-dimensional analysis and federated query capabilities were used to accelerate the query display performance of the digital platform, abandoning the original T+1 report. , implementing instant query on top of Apache Doris.

future plan

Currently, the analysis platform built based on Apache Doris is mainly used for Xiaoetong live broadcast business to provide services for user-related data. After the introduction of Apache Doris, the platform architecture has achieved unified writing and checking exports, greatly improved timeliness, and expanded multi-dimensional analysis scenarios for user data. At the same time, the support of Doris has also given the back-end system key advantages such as stability, scalability, and security. These breakthroughs have further improved the basic technical capabilities of the platform, and also promoted further growth in the number of users and live broadcast benefits.

In the future, we hope to continue to expand the usage scenarios of Apache Doris and hand over course analysis, user log analysis, etc. to Apache Doris to provide unified data analysis services to the outside world. At the same time, we also hope to continue to explore the new features of Apache Doris 2.0 version to further improve cluster stability, including:

Pipeline execution engine and cost-based query optimizer : When making user live broadcast lists, there are many complex after-sales scenarios that require manual optimization. With the Pipeline execution engine and query optimizer, there is no need to manually adjust SQL, making execution performance better. excellent;
CCR cross-cluster data synchronization : improve the availability of online service data through automatic synchronization of user database tables in multiple clusters;
Inverted index : Use the inverted index function of Apache Doris version 2.0 to enrich the existing index structure and implement capabilities such as fuzzy query, equivalent query, and range query to further accelerate query response speed.

Special thanks to the SelectDB technical team for responding promptly to any problems encountered during use, which has reduced a lot of trial and error costs for us. In the future, we will also actively participate in Apache Doris community contributions and activities, and progress and grow together with the community!

Application practice of Apache Doris in Xiaoetong