Preface
The technical team of Meituan's takeaway data warehouse is responsible for supporting daily business operations and analysts' daily analysis. Due to the high data production cost and low query efficiency brought about by the characteristics of the takeaway business, they introduced the Apache Doris engine to optimize the production plan The balance of low-cost production and efficient query. And analyze the applicability of MOLAP model based on Kylin and ROLAP model based on Doris engine in different business scenarios. I hope to be able to inspire or help everyone.
This article focuses on the improvement and thinking of the data warehouse production architecture with the Doris engine as the "engine". In the open source environment, various data engines are in full bloom, but due to the complexity and diversity of business, there is no engine that can adapt to all business scenarios. Therefore, we hope to provide some experience for everyone through our business practice and thinking. . Meituan's takeout data warehouse technical team is committed to maximizing the efficiency of data application, taking into account the minimization of R & D, production and operation and maintenance costs, and building continuous improvement of the data warehouse capacity. You are also welcome to give us more suggestions.
Application Status of Data Warehouse Interactive Layer Engine
At present, the scale of the Internet business has become larger and larger. Whether it is a business production system or a log system, basically a data warehouse is built based on the Hadoop / Spark distributed big data technology ecology, and then the data is appropriately layered and processed ,management. At the data application interaction level, due to the requirements of timeliness, the final display query of the data still needs to be supported by the DBMS (MySQL) and MOLAP (Kylin) engines. As shown below:
Aggregate data interaction
The most typical scenario of business team's daily business analysis is custom query in various dimensions. Faced with such flexible and variable WYSIWYG application scenarios, Meituan platform uses Kylin as the company's main MOLAP engine. MOLAP is a pre-calculated production. It performs well in incremental business and preset dimension analysis scenarios, but it has a huge production cost in changing dimension scenarios. For example, if you use the latest merchant type to backtrack the performance of the merchant for the past three months, you need to recalculate the three-month Cube, which will take a few hours to calculate the historical data of nearly TB. In addition, to deal with non-predetermined dimensional analysis, the MOLAP model needs to be recalculated for adaptation and iterative work is also required.
Detailed data interaction
In addition to macro data, business analysis is also just a need for detailed data query. Usually, you will choose relational DB such as MySQL as a quick query for detailed data, but when the business grows faster, you will soon encounter performance bottlenecks and high maintenance costs. For example, the maintenance cost of operations such as synchronization of large amounts of data, new fields, and historical data updates are very high.
Takeaway operation business characteristics
Meituan ’s mission is to "help everyone eat better and live a better life." The take-out business provides food delivery services for everyone and connects merchants and users. This is a labor-intensive business. The take-out business has an operation team of tens of thousands of people to serve millions of merchants across the country, and takes the "business circle" as a unit. Serving businesses in the "business circle". "Business district" is the smallest level in the dimension of an organization, which is derived from the characteristics of the take-out organization. "Business district" and its upper-level organizations are a changing dimension. When the boundaries of the "business district" change, it leads to In the incremental business production mode, the traceback of historical data loses its reference significance. In all business scenarios that present organizational data, changes in the organizational structure are an unavoidable technical issue. In addition, there are also problems of changing dimensions in other dimensions such as merchant category and type. As shown below:
Challenges facing data production
The data is exploding, and the latest dimension is used to retrospectively calculate historical data. The following problems exist in Kylin's MOLAP mode:
- Historical data is refreshed daily, losing the meaning of increment.
- A large amount of historical data is traced daily, and one billion + historical data is traced back.
- Data calculation takes 3 hours +, storage 1TB +, consumes a lot of computing storage resources, and seriously affects the stability of SLA.
- The actual usage rate of a large amount of pre-calculated historical data is low, and 80% of the history backtracking in actual work is concentrated in the past month or so, but in order to respond to all demand scenarios, the business requires calculation of more than half a year of history.
- The query of detailed data is not supported.
Solution: Introduce MPP engine, data is used now
Since the pre-calculation cost of the historical data of change dimension is huge, the best way is to use the current calculation, but the current calculation requires strong parallel computing capabilities. There are three forms of OLAP implementation: MOLAP, ROLAP, and HOLAP. MOLAP uses Cube as its performance form, but the calculation and management costs are relatively high. ROLAP needs the support of a powerful relational DB engine. For a long time, due to the limited data processing capabilities of traditional relational DBMS, the ROLAP model has been greatly limited. With the mature application of distributed and parallelized technologies, MPP engines have gradually demonstrated powerful high-throughput and low-latency computing capabilities. There are not many engines known as "100 million-second open", and the ROLAP model can be better extended. Considering the practical application of the business alone, the performance can cover many application scenarios and have the possibility of application when the on-site calculation of tens of millions of related queries is calculated in seconds. For example: ROLAP on-site calculation of daily data volume, calculation of weekly and monthly trends, and browsing of detailed data can all be handled well.
The following figure is a comparison of the application solutions in MOLAP mode and ROLAP mode:
Disadvantages of the MOLAP model
- The application layer model is complex, and more model preprocessing is required according to business needs and Kylin production needs. In this way, the utilization rate of the model is relatively low in different business scenarios.
- The Kylin configuration process is cumbersome and requires configuration model design and appropriate "pruning" strategies to achieve a balance between computational cost and query efficiency.
- Since MOLAP does not support the query of detailed data, in the "summary + detailed" application scenario, the detailed data needs to be synchronized to the DBMS engine to respond to the interaction, which increases the operation and maintenance cost of production.
- More pretreatment is accompanied by higher production costs.
Advantages of ROLAP model
- The application layer model design is simplified, and the data can be fixed at a stable data granularity. For example, the star model of the merchant granularity, and the reuse rate is relatively high.
- The business expression of the App layer can be encapsulated through the view, which reduces data redundancy, and at the same time improves the flexibility of the application and reduces the cost of operation and maintenance.
- Also supports "Summary + Details".
- The model is lightweight and standardized, which greatly reduces production costs.
In summary, in the application scenarios of changing dimensions, non-preset dimensions, and fine-grained statistics, using the ROLAP mode driven by the MPP engine can simplify model design and reduce the cost of pre-calculation. Support a good real-time interactive experience.
Application scenario adaptation problems under dual engines
The architecture uses MOLAP + ROLAP dual engine mode to adapt to different application scenarios, as shown in the following figure:
Technical trade-offs
MOLAP : Through pre-calculation, provide stable slice data, realize multiple queries and one calculation, reduce the calculation pressure during query, ensure the stability of the query, and is the best path of "space for time". It implements a deduplication algorithm based on Bitmap, supports real-time statistics of deduplication indicators in different dimensions, and has high efficiency. ROLAP : Based on real-time massively parallel computing, the requirements on the cluster are high. The core of the MPP engine is to improve the parallel computing capability by dispersing data to achieve the distribution of CPU, IO, and memory resources. In the case where the current data storage is dominated by disks, the larger disk IO required by Data Scan and the high CPU caused by parallelism are still shortcomings in resources. Therefore, high-frequency large-scale summary statistics, concurrency capabilities will face greater challenges, which depends on the parallel computing capabilities of the cluster hardware. Traditional deduplication algorithms require a lot of computing resources, and real-time large-scale deduplication indicators are a huge challenge for CPU and memory. At present, the latest version of Doris already supports the Bitmap algorithm, and with the pre-calculation, it can solve the deduplication application scenario well.
Business model adaptation
MOLAP : When the business analysis dimension is relatively solid, and when the historical state is available, incremental production is performed according to time, the processing cost is linearly increasing, and the data is processed to a coarser granularity (such as organizational unit), reducing the amount of result data and improving Interactive efficiency. As shown in the figure above, from the A model to the B model, using Kylin is a good choice.
ROLAP : When the business analysis dimension is flexible or specific to the latest state (as shown in the model A above, always use the latest business organization attribution to view history), it is costly to pre-calculate the backtracking historical data. In this scenario, stabilizing the data at the granularity of the merchant, performing retrospective analysis of historical data through on-site calculations, and realizing current and current calculations can save huge costs of pre-calculation and bring greater application flexibility. In this case, it is suitable for the ROLAP production mode supported by the MPP engine.
Selection of MPP engine
At present, there are many OLAP engines that are more concerned about open source, such as Greenplum, Apache Impala, Presto, Doris, ClickHouse, Druid, TiDB, etc., but there is no introduction of practical cases, so we do not have much experience to learn from. Therefore, based on the needs of our own business, we started from the cost of engine construction, and based on the company's technology ecological integration, integration, ease of use and other comprehensive considerations, as a basis for selection, in the end, our platform department chose to enter in 2018. Doris in the Apache community.
Doris Introduction and Features
Doris is an OLAP engine based on MPP architecture, which mainly integrates the technologies of Google Mesa (data model), Apache Impala (MPP Query Engine) and Apache ORCFile (storage format, encoding and compression).
The system architecture of Doris is as follows, which is mainly divided into two components, FE and BE. FE is mainly responsible for query analysis, compilation, optimization, scheduling, and metadata management; BE is mainly responsible for query execution and data storage. For more technical details about Doris, please refer to its official documentation .
Features of Doris:
- It also supports high concurrency query and high throughput Ad-hoc query.
- It also supports offline batch import and real-time data import.
- It also supports detailed and aggregate queries.
- Compatible with MySQL protocol and standard SQL.
- Support the intelligent query routing of Rollup Table and Rollup Table.
- Support better multi-table Join strategy and flexible expression query.
- Support Schema online change.
- Support Range and Hash secondary partition.
Doris's application efficiency in takeout warehouse
The above picture is our assessment of project returns in an analysis project transformation. Overall, when the query efficiency is unchanged, production energy consumption and storage costs have greater benefits.
With a Doris environment of 20 BE + 3FE, the efficiency and performance are as follows:
- Supporting more than dozens of data analysis products, the overall response reaches ms level.
- It supports millions and tens of millions of large table association queries, and at the same time, the snowflake model associated with the dimension table is optimized. The Colocate Join feature is optimized to achieve second-level response.
- The daily level is calculated on the spot based on the merchant's details, while satisfying the summary and drill-down detailed queries. The query time can basically be controlled in seconds.
- 7-day trend analysis, 2 ~ 3 seconds. Due to the large amount of data, the query performance differs according to the size of the cluster. However, when the amount of data is large, more cluster resources are mobilized, so the concurrent performance of MPP is limited by the performance of the cluster. The general principle is that services with high concurrency need to strictly control the query time (basically in the millisecond level). For services with low concurrency, larger queries are allowed, but the affordability of the cluster must also be considered.
- Through one year of application and continuous improvement and upgrade of Doris, Doris' high reliability, high availability, and high scalability have also been further verified, and the service is stable and reliable.
Applications in quasi-real-time scenarios
Most offline business analysis is based on T + 1 offline data. However, in the context of marketing activities, the delivery team often needs the real-time data of the day to monitor and analyze the business changes. Usually, real-time streaming computing is used.
The real-time business monitoring of takeout has the following characteristics:
- To avoid the impact of minute-level production fluctuations, the quasi-real-time data of 10 and 15 minutes on the business can meet the analysis needs.
- The real-time data needs to be compared with the offline data on a daily basis and on a weekly basis.
- Order business needs event time, experience business needs production time, business alignment logic is complex.
- The demand for different business lines varies greatly, and the indicators need to be well scalable.
Due to the complexity of the business, the alignment of many business calibers needs to be considered in real-time stream calculation. The business ER model has a high development cost and a large resource consumption in the confluence process. By designing a quasi-real-time production data warehouse based on Doris, it can be flexible Realize business micro-batch processing, and development and production costs are relatively low. The following is a quasi-real-time data warehouse architecture design based on Doris, which is a typical real-time Lambda production architecture:
The realization of a quasi-real-time computing solution requires the support of the following capabilities:
Real-time write capability : Currently supports Kafka To Doris second-level delay. Reliability and stability construction still need to be further improved. Engine construction : short and fast calculation + efficient storage performance. At present, there is still room for improvement in the performance of the Doris engine. There will be major improvements in 2020. With the subsequent launch of Page Cache, memory table and other capabilities, IO will no longer be behind, and the concurrency capability will be greatly improved. Reliable scheduling capabilities : provide 5, 10, 15, and 30 minutes of scheduling support capabilities. Lambda architecture simplifies : real-time data and offline data are better integrated in Doris, supporting applications flexibly. Efficient OLAP interaction : Support flexible query and access to the business. The business layer directly encapsulates the multi-dimensional model of the summary layer through logical encapsulation through views, which improves development efficiency and reduces operation and maintenance costs.
Compared with the window calculation in Storm and Flink, the advantages of quasi-real-time DB micro-batch:
Important improvement of Doris engine in Meituan
Transitive optimization of join predicate pushdown
As shown in the figure above, for the following SQL:
select * from t1 join t2 on t1.id = t2.id where t1.id = 1
复制代码By default, the open source version of Doris will perform a full table scan on the t2 table, which will cause the above query to time out, which will lead to the first batch of applications for the takeaway business on Doris not going online.
So we implemented the first optimization in Doris: the transitive optimization of Join predicate pushdown (called Constant Propagation in MySQL and TiDB). The transitive optimization of Join predicate pushdown refers to: based on the predicates t1.id = t2.id and t1.id = 1, we can infer the new predicate t2.id = 1, and push down the predicate t2.id = 1 Go to the Scan node of t2. In this way, if the t2 table has hundreds of partitions, the query performance will be improved by tens or even hundreds of times, because the amount of data that the t2 table participates in Scan and Join will be significantly reduced.
Query execution multi-instance concurrency optimization
As shown in the figure above, Doris only generates 1 execution instance for each operator on each node by default. In this case, if the amount of data is large, the operator of each execution instance needs to process a large amount of data, and the CPU, IO, and memory resources of the cluster cannot be fully utilized.
An optimization method that is easier to think of is that we can generate multiple execution instances for each operator on each node. In this way, each operator only needs to process a small amount of data, and multiple execution instances can be executed in parallel.
The following figure shows the optimization effect with concurrency set to 5. You can see that for various types of queries, there will be a 3 to 5 times increase in query performance:
Colocate Join
Colocate Join (Local Join) is a concept opposite to Shuffle Join and Broadcast Join, that is, the data of the two tables is advanced according to Join Key Shard, so that there is no overhead of data network transmission when Join is executed, and the two tables can be directly joined locally.
The key points of the realization of Colocate Join in Doris are as follows:
- Ensure data locality when importing data.
- Ensure data locality during query scheduling.
- After data balance, the data locality is guaranteed.
- Query Plan changes.
- Colocate Table metadata persistence and consistency.
- The granularity of Hash Join changes from Server granularity to Bucket granularity.
- Colocate Join condition judgment.
For more implementation details of Doris Colocate Join, please refer to "Apache Doris Colocate Join Principles and Practice".
For the following SQL, the performance comparison of Doris Colocate Join and Shuffle Join under different data volumes is as follows:
select count(*) FROM A t1 INNER JOIN [shuffle] B t5 ON ((t1.dt = t5.dt) AND (t1.id = t5.id)) INNER JOIN [shuffle] C t6 ON ((t1.dt = t6.dt) AND (t1.id = t6.id)) where t1.dt in (xxx days);
复制代码Bitmap exact deduplication
The way Doris used to achieve accurate deduplication was calculated on-site. The implementation method is similar to Spark and MapReduce:
For the SQL for calculating PV in the above figure, Doris will calculate according to the following figure when calculating, first according to the page column and user_id column group by, and finally Count:
Obviously, in the above calculation method, when the amount of data is getting larger and larger, reaching billions or tens of billions, the IO resources, CPU resources, memory resources, and network resources used will become more and more, and the query will become Slower and slower.
So we added a new Bitmap aggregation indicator in Doris. When data is imported, the data of the same dimension column will use Bitmap aggregation. With Bitmap, the way to calculate the exact deduplication in Doris is as follows:
It can be seen that after using Bitmap, the previous PV calculation process will be greatly simplified, and the IO, CPU, memory, and network resources during on-site query will be significantly reduced, and will no longer increase linearly with the data size.
Summary and thinking
In the business practice of takeaway operation analysis, due to the complexity of the business and the different application scenarios, no single data production solution can solve all business problems. The development of database engine technology provides us with more means to improve data construction solutions. Practice has proved that the ROLAP mode driven by the Doris engine can better deal with scenarios such as summary and details, historical traceback of changing dimensions, flexible application of non-preset dimensions, and quasi-real-time batch processing. The MOLAP model based on Kylin is still important in dealing with incremental business analysis, solidifying dimensional scenarios, and swapping space for time through pre-calculation.
In terms of business, through the successful practice of take-out of several warehouses Doris and cross-BG communication, Meituan already has more teams to understand and try to use the Doris program. Moreover, with the joint efforts of the platform students, there is still much room for improvement in engine performance. I believe that the ROLAP model driven by the Doris engine will bring greater benefits to the business team of Meituan. Judging from the current practical results, it has a tendency to replace engines such as Kylin, Druid, and ES.
At present, database technology is advancing rapidly. Recently, Bairui Data released RapidsDB v4.0, a full-memory distributed database that supports TB level millisecond response (processing 100 billion data can achieve millisecond response). It is foreseeable that the advancement of database technology will greatly improve the hierarchical management and application support efficiency of the data warehouse, and the business will become "defined and visible", and will greatly enhance the value of the data.
References
About the Author
- Zhu Liang is a data warehouse engineer for Meituan.
- Kaisen, Big Data Engineer of Meituan, Apache Kylin Committer.
Job Offers
Meituan Takeaway Data Intelligence Group has long recruited engineers for data warehouse, data mining, machine learning, computer vision, search recommendation algorithms, and coordinate Beijing. Interested students are welcome to send their experience to: [email protected] (email title indicates: Takeaway Data Intelligence Group)
Read more technical articles, please scan the code to follow the WeChat public account-Meituan technical team!