01
minimalist architecture
In terms of design, Doris integrates Google Mesa's data storage model, Apache's ORCFile storage format, Apache Impala query engine and MySQL interaction protocol. It is a leading design product with advanced technology and advanced architecture.
In terms of architecture, Doris has only two types of processes: one is FE , which can be understood as the management node of Doris, which is mainly responsible for the access of user requests, the analysis of query plans, the storage of metadata and cluster management related work; the other is BE is mainly responsible for data storage and execution of query plans. Both types of processes are scalable horizontally. In addition, Doris does not rely on any third-party systems (such as HDFS, Zookeeper, etc.). This highly integrated architecture design greatly reduces operation and maintenance costs.
FE nodes include three roles: Leader , Follower , and Observer . By default, a cluster can only have one Leader, and there can be multiple Followers and Observers. Among them, the Leader and Follower form a Paxos selection group. If the Leader goes down, the remaining Followers will automatically select a new Leader to ensure high write availability. Observer synchronizes Leader's data, but does not participate in elections. If only one FE is deployed, the FE is the Leader by default.
The FE node mainly includes a storage management ( Store Manager ) module, a state management ( State Store ) module, a coordination ( Coordinator ) module, a metadata ( StoreMeta ) module and a metadata cache ( StoreMeta Cache ) module. The storage management module is responsible for managing all metadata information, including database, table information, tablet information, tablet copy information, etc. The storage management module is also responsible for managing user authority information (that is, user authentication information and authorization information) and data import tasks. The status management module is responsible for managing the survival status information of all BE processes, querying load information and other non-persistent information, and providing publish and subscribe interfaces. The coordination module is responsible for receiving requests from users, then parsing statements, generating execution plans, and scheduling the execution plans according to the current cluster status. The metadata module is responsible for reading and writing metadata, and only the FE Leader has this permission. The metadata cache module is responsible for synchronizing metadata for statement parsing and execution plan generation, mainly the permissions of Follower and Observer roles.
BE nodes can be expanded infinitely, and the roles of all BE nodes are equal. When the cluster is large enough, some BE offline will not affect the cluster to provide services. The BE node is mainly composed of a storage engine (Store Engine) and a query executor (Query Executor). The storage engine is responsible for managing the local tablet data of the node, sending or receiving data to form a copy, and regularly merging and updating multiple versions of data to reduce storage usage. The storage engine is also responsible for receiving data reading requirements and batch data import requirements from query executors. When a query is executed in an MPP cluster, it will be split into a tree-like execution tree. The execution of this tree is coordinated by the Coordinator. The leaf nodes of the tree are also called plan fragments (PlanFragment), and each PlanFragment is assigned to a BE node. The query executor to execute, this is the role of the query executor module.
02
Simple to use
Doris is not only simple in structure, but also very simple in development and use. For an OLAP database, performance is not the whole of the database, and ease of use is the key to deciding whether to continue to use it. Doris has always taken the user's ease of use as the starting point from the beginning of system design.
From the perspective of the whole cycle of data analysis, it can generally be simply summarized into four aspects, from data modeling→data import→user analysis→continuous use and maintenance and upgrades, the ease of use of Doris is everywhere.
In terms of data modeling, Doris supports three models: Aggregate , Unique and Duplicate , which can meet various application scenarios in the OLAP field. At the same time, the Doris table creation statement only adds some features of distributed systems compared to MySQL, such as distribution keys and bucket numbers. Users who have experience in using distributed databases are very easy to understand and operate.
In terms of data import, Doris provides a variety of data import solutions (as shown in Figure 2), which can be selected for different data sources, while providing atomicity guarantees during the data import process. Whether you use Broker Load for batch import or use INSERT statement for single import, it is a complete transaction operation. The import transaction can ensure that the data in a batch takes effect atomically, and there will be no partial data writing.
At the same time, each import job will generate a Label, which is used to uniquely distinguish an import task in the database. Label can be specified by the user, and some import functions will also be automatically generated by the system. Label is used to ensure that the corresponding import job can only be successfully imported once. When a successfully imported Label is used again, it will be rejected and the error Label already used will be reported. Through this mechanism, the data consumption side can realize At-Most-Once semantics. If combined with the At-Least-Once semantics of the upstream system, the Exactly-Once semantics of end-to-end data import can be realized.
In terms of SQL development, Doris supports standard SQL language and is compatible with MySQL in dialect . Whether it is simple single-table aggregation, sorting and filtering operations, or complex multi-table associations, subqueries, window functions, etc.,Doriscan be easily completed throughSQL, which greatly reduces the cost of user migration and use. High-throughput ad hoc queries such asAdhocETLscenarios are alsoDoris' strengths. Doriscan also support complexSQLsyntax, includingGrouping Set, and can alsocustomize and expand functionsUDForUDAFForterabytesof data,Doriscan partially replaceHive, allowing users to meet all requirements in one database.
In terms of tools, Doris has implemented MySQL protocol compatibility in the FE module , which is convenient for users to use standardMySQLclients or class libraries in various languages to connect, and supports various tools very well. In terms of database development[3] [4] , we can seamlessly useDBeaver,DataGrip, andNavicat; in terms of programming applications[5] [6] ,Dorisfully supportsMySQL'sJDBCandODBCinterfaces, and can supportC,Python,Java,Shelland other development languages; inBIapplications,Dorissupports FanruanBI, GuanyuanBI, YonghongBI,Tableauand other agile BI software; in terms of ETL scheduling, Doris supports Kettle , DolphinScheduler and other mainstream software.
In terms of cluster reliability, Doris metadata uses the mode of memory storage + checkpoint + mirror log file , and usesthe BTBJE(similar toRaft) protocol to achieve high availability and high reliability of metadata. Dorisinternally manages multiple copies and automatic repair of data to ensure high availability and reliability of data. When some servers go down, the cluster can still run normally and the data will not be lost. Dorisdeployment has no external dependencies, and only needs to deployBEandFEmodules to build a cluster. Dorissupports online change of table mode (adding and subtracting columns, creatingRollup), which will not affect the current service, and will not block read and write operations, because such operations are performed asynchronously.
In terms of cluster expansion and contraction, based on its own distributed management framework, Doris can automatically manage the distribution, repair and balance of data copies. For example, if the copy is damaged,Doriswill automatically detect and repair it. For node expansion and contraction, only oneSQLcommand is needed to complete it, andDoriswill automatically perform data fragmentation balancing. The whole process does not affect system services at all, and does not require any additional operations by operation and maintenance personnel.
In terms of cluster upgrade, the upgrade method of Doris is extremely simple. It only needs to replace the binary program and restart the cluster in a rolling manner. In terms of design,Dorisis fully forward compatible, so the new version can also be verified and tested through grayscale upgrades. Some of Doris's own failure retry and fault routing functions also greatly reduce the impact of errors that occur during the upgrade process on the business.
03
feature rich
Doris provides a very rich set of functions to help businesses adapt to different application scenarios. The following highlights some of Doris's characteristic functions.
The first is the partition and bucket pruning function . Doris supports two levels of data division: the first level is Partition, which supports the division of Range and List. The second layer is Bucket bucketing, which divides the data horizontally through Hash, and the data fragmentation tablets are evenly dispersed in the cluster.
Using the bucket pruning function, Doris can fix queries to a very small number of shards, thereby significantly reducing the consumption of system resources by a single query and improving the overall concurrent query capability of the cluster. In high-concurrency query scenarios, a single Doris node can support query requests of thousands of QPS.
The second is a reasonable cache function . Doris also supports SQL-level and Partition-level query caching. Among them, the SQL-level cache uses the Hash value of the SQL statement as the key to directly cache the SQL results, which is very suitable for scenarios where the update frequency is not high but the query is very frequent. The Partition-level cache will intelligently cache the result data of different partitions in the SQL result, and subsequent queries can use the data of the cached partition plus the real-time query data of the new partition to obtain the final result, thereby reducing the real-time cost of duplicate data. Query requirements to reduce the consumption of system resources.
Again , Doris supports the Bitmap data type . This data type uses bitmaps to store integer data, and can perform some collection operations through bitmaps. Bitmap can be applied to high cardinality accurate deduplication scenarios. The traditional real-time deduplication algorithm needs to build a Hash table in the memory to deduplicate the data. When the cardinality is very high, it will occupy a large amount of memory. With Bitmap, the numerical type can be converted into 0 and 1 on the bitmap, thereby greatly reducing the memory overhead, and for deduplication calculation, it is only necessary to calculate the number of 1 after the intersection of multiple Bitmaps, so as to achieve Perform fast high-radix exact deduplication calculations with limited memory overhead.
In the user portrait scenario, Bitmap is used to store user IDs, and crowd packages with different tag combinations can be quickly obtained through bitmap set operations. At the same time, Doris also has many built-in functions related to Bitmap, which are used to calculate the funnel, retention, etc. For example, through the intersect_count() function, the user retention can be easily calculated.
And finally the materialized view . Materialized view is also one of the core features of Doris . The materialized view is to store the pre-calculated (according to the defined SELECT statement) data set in a view table that is transparent to the user and has real data. The materialized view is mainly to meet the needs of users who can not only analyze the original detailed data in any dimension, but also quickly analyze and query the fixed dimension, and analyze the detailed and aggregated data from a unified perspective.
In Doris, users can use the detailed data model to store detailed data, and then select arbitrary dimensions and indicators to create aggregated views on the detailed data, such as SUM, MIN, MAX, COUNT, etc. Doris will ensure that the data in the schedule and the materialized view are completely consistent. If the physical table data is imported or deleted, the materialized view will be automatically updated to ensure data consistency between the original table and the materialized view table. At the same time, the materialized view is transparent to user queries, and Doris will automatically match the most suitable materialized view for query according to the pattern in the query statement. Through the materialized view function, users can unify the detail and aggregation models on one table, and speed up the query response of certain fixed modes.
Doris also supports data updates based on primary keys. Through the Unique data model, users can update data based on the primary key. At the implementation level, Doris uses the Merge-on-Read method to provide updated data. In addition, users can also use the REPLACE_IF_NOT_NULL aggregation method to meet the needs of partial column updates. For the Unique data model, Doris also supports friendly Update operations.
Based on the Unique model, Doris also uses functions such as Marked Delete and Sequence Column to realize the synchronization operation of the updated data of the upstream transaction database, and can not only guarantee the atomicity of transactions, but also the order of data synchronization.
04
open source
Another particularly important feature of Doris is that it is completely open source and open . As a project of the Apache Foundation, Apache Doris complies with the Apache License 2.0. Apache License 2.0, as the most mainstream open source protocol, is recognized by OSI as "the following OSI-approved licenses are popular, widely used, or have strong communities".
For the specific content of Apache License 2.0, you can check it on the Apache official website. In short, the distribution is completely free, the project code is allowed to be modified, and it is allowed to be republished as open source or commercial software. Once the authorization is permanently valid , an agreement with the original code and a patent statement are required in the modified code or the code derived from the source code wait. This is an extremely friendly agreement for any commercial company and open source user.