Abstract: This article is compiled from Zhou Yueyue, architect of OceanBase, sharing in the real-time lake warehouse session of Flink Forward Asia 2022. The content of this article is mainly divided into four parts:
Interpretation of key technologies of distributed database OceanBase
Ecological docking and typical application scenarios
Practice of OceanBase X Flink in the game industry
future outlook
Click to view the original video & speech PPT
1. Interpretation of key technologies of distributed database OceanBase
As a purely self-developed domestic distributed database after 12 years, from product project approval to core transaction business launch, OceanBase has firmly moved towards a distributed architecture from the 1.0 era, and the product has begun to be implemented in Alipay and support core business.
With the further enhancement of product capabilities, the OceanBase 2.0 era evolved from a KV storage system to a native distributed database with distributed transactions and strong consistency of multiple copies, and began to serve external corporate customers, including the Internet, finance, securities and other industries .
In the 3.0 era, with the improvement of HTAP capabilities, hybrid engines and hybrid deployment solutions attract more overseas enterprise customers to use. With the release of version 4.0, OceanBase proposes a stand-alone distributed integrated architecture to help enterprises miniaturize and provide public cloud services.
The integrated architecture of OceanBase can be summed up with three keywords: Paxos protocol, shared-nothing architecture, and partition-level high availability.
By default, data is stored in multiple copies, the concept of multiple copies. Strong data consistency is guaranteed between replicas through the Paxos protocol. Through the multi-copy + Paxos protocol, the high availability of the database system is guaranteed.
Each OBServer node in the system has both computing and storage capabilities. There is no single-point bottleneck in the whole system, and multiple points can be read and written. When the cluster expands and shrinks, data is migrated using partitions as the basic unit to automatically achieve load balancing.
As a system that bears the lifeblood of an enterprise, high availability of the database is crucial to the enterprise. OceanBase's typical three-copy deployment solution based on the Paxos protocol ensures that data will not be lost and services will not be interrupted when a single machine, computer room, or city fails.
Cost reduction and efficiency increase is an eternal topic for enterprises, so how to reduce hardware costs through technical means is a concern of every enterprise. When data is written to OceanBase, it is first written into the memory, and when the conditions are met or the set threshold is triggered, the data will be refreshed to the disk.
Therefore, the full amount of data in OceanBase consists of disk baseline data and memory incremental data, so OceanBase is sometimes called a quasi-memory database. In terms of data compression, the LSM tree data structure used by OceanBase has a corresponding compression algorithm at each layer. This type of compression is called general compression.
On the basis of general compression, OceanBase self-developed a set of compression methods (encoding) for mixed row and column encoding of the database, which will further compress the data. The storage space is further reduced on the basis of general compression. Under the same conditions, compared with Oracle, the storage cost of OceanBase is only about 1/3 of the former.
In traditional database solutions, such as the most commonly used MySQl database, multiple businesses are generally split into multiple databases. Physically isolate. Avoid single business exceptions affecting the entire business system. With the rapid growth of business, operation and maintenance personnel need to operate, maintain and manage multiple sets of environments, and the cost is high.
In OceanBase, in the case of sufficient resources, only need to create new tenants to access new services, and achieve resource isolation and data independence between services. The resource isolation scheme between tenants ensures that one environment can carry multiple sets of services, and the workload of operation and maintenance personnel is greatly reduced.
HTAP is a topic that has been constantly mentioned in recent years, so is HTAP a false proposition? In fact, HTAP does not appear out of thin air. The reality is that users have real business needs and actual scenarios.
In the previous solution, the data generated by the TP business was synchronized to some analytical products through tools to perform tasks such as data analysis and batch running. This involves the splicing of multiple systems, as well as the transfer and storage of multiple copies of data.
At present, the HTAP that everyone agrees on is also the HTAP that the OB thinks: that is, while doing a good job in TP, taking into account and improving the analysis ability. In this concept, there are two core points, namely a piece of data and a set of systems. The data can be processed in one system, and there is no need to synchronize and transfer again.
In addition to a set of SQL engines to meet the needs of TP and AP, OceanBase can also flexibly implement various read-write separation strategies through multiple copy types and weak consistency according to the user's read-write separation requirements, ensuring that the original business will not be changed. The cost is 0, which can meet the needs of users.
As a distributed database, scalability is the most important capability. Under the integrated architecture of OceanBase, the cluster nodes are equal, each node has computing and storage capabilities, and can be expanded and reduced online at the same time. Each node can read and write. In theory, the performance of the cluster increases linearly with the expansion of nodes.
Not long ago, OceanBase released version 4.0, introducing a stand-alone distributed integrated architecture. Distributed architecture is more used in business scenarios with large data volume and scale, and in these scenarios, the distributed advantages can be better utilized.
For enterprise users whose business data volume is not large enough, or whose current data volume is not large, the distributed solution requires too much resources, so it is not suitable for small and medium business volume scenarios. At the same time, a stand-alone or lightweight architecture is more suitable for this type of business. The stand-alone integrated architecture solution, while using a stand-alone machine, can change the stand-alone machine into a distributed architecture as the data scale grows in the future, fully meeting the needs of business development.
2. Ecological docking and typical application scenarios
As a representative product in the field of real-time analysis, Flink is used by many OceanBasse community users and used in real-time data warehouse business scenarios. According to the needs of community users, we have docked and adapted Flink and its surrounding ecological tools, such as ChunJun.
Users use Flink and corresponding ecological tools to allow data to flow freely in different systems. For example, the upstream source MySQL or OceanBase data is synchronized to the downstream OceanBase, Kafka and other destinations.
In the OceanBase community, many users use OceanBase+Flink to solve practical problems encountered in production. Typical application scenarios include:
Scenario 1, real-time data writing and data cleaning, which is also the most used scenario. When data is stream-written to the downstream, it is not only necessary to ensure the real-time performance of the write, but also there may be problems such as data format cleaning and conversion. Therefore, Flink can realize the real-time writing of data to downstream databases such as OceanBase, etc., and at the same time Actions such as data cleaning can be performed during the writing process.
Scenario 2: Broaden the data flow. Multi-table join and association with dimension table and fact table are the most common scenarios. In the figure above, the business data source will continuously generate a data stream, and perform join operations with the dimension tables in OceanBase to widen the data stream and generate a large wide table. Finally, the data is written into a result set and stored in a database system, such as OceanBase.
Scenario 3: Building a materialized view. When business data is continuously written to OceanBase, the data in the table is constantly changing. At this time, when performing some query operations, such as aggregation query, a single piece of newly added data will trigger query calculation. When the data scale involved in the query is large and the data is updated frequently, the query performance will be unsatisfactory.
After using Flink, the data stream is converted into a dynamic table and aggregated continuously. Store the generated result set downstream, such as OceanBase, etc. Users only need to query the result set to get the required data, without performing aggregation operations every time, and the performance improvement is very obvious.
Scenario 4: secondary processing of data. With the popularity of distributed solutions, enterprises use the scalability of distributed databases to store raw data in big data scenarios in databases, such as various indicator data.
When it is necessary to reprocess the indicators of the original data, with the help of Flink's real-time synchronization capability, the indicator data is reprocessed during the synchronization process, and the processed data is written back to OceanBase for business use. At the same time, the processed data can be processed again as the source, which is very flexible to use.
3. Practice of OceanBase X Flink in the game industry
As enterprises pay more and more attention to the value of data, the freshness of data is crucial, and enterprises need to be able to observe changes in data in real time. For example, in the delivery of express delivery, enterprises need to grasp the operation status of express delivery in the whole process from the user's order to the user's receipt in real time, timely discover possible problems in each link and quickly solve them, so as to improve operational efficiency and user experience.
During the prime time of traffic, corporate decision makers need to always pay attention to the situation of popular advertising positions, adjust advertising placement in a timely manner, and maximize the value of advertising positions.
In the field of big data real-time data warehouses, data warehouses provide data-supported strategies for the decision-making process of enterprises. The Lambda architecture is an earlier data warehouse solution that uses two architectures, stream processing and batch processing, for data processing. The data warehouse architecture of a game company is shown in the figure: offline processing is handed over to Hive, and real-time analysis is completed by Click House.
Hive is a Hadoop-based data warehouse tool that can perform data organization, special query and analysis processing on data sets stored in HDFS. Spark is an open-source cluster computing system based on in-memory analysis and computing. The purpose is to make data analysis faster. Hive+Spark complement each other. And Spark+Click House is written into Click House through Spark micro-batch, to play the analysis ability of Click House.
There are three typical scenarios in the game industry:
-
Scenario 1: Query the user ID through the ID number. When a user registers, the system needs to use the ID number information to go to each platform to check whether there is already registration information or multiple IDs. If there is already registration information, the user is prompted to log in.
-
Scenario 2: Check the advertising channel through the user ID. After the user registers, the third-party channel provider needs to get a callback on whether the attribution is correct, such as whether the user registered from this channel has been hacked.
-
Scenario 3: View real-time advertising effects. When the game anchor promotes the game, he needs to see the data of the game's clicks, downloads, installations, registrations, creation of characters, channels and other indicator information in real time. Corresponding to the business level, it involves the associated operations of 7 tables.
In scenarios 1 and 2, it takes 66 seconds to analyze using Click House; in scenario 3, it takes more than 20 minutes to complete the query in the Hive solution.
Combining the characteristics of business testing and architecture, the current challenges mainly include the following four aspects:
- Real-time performance is not enough. Under the Hive architecture, it takes 30 minutes for data to be imported and visible, while ClickHouse also takes one minute.
- Insufficient consistency. I believe that anyone who has used the Lambda architecture knows that the data of ClickHouse and Hive often "fight", and the data calculated by the two are inconsistent. It is necessary to deduplicate the calculations, but even after repeated processing, there are still data inconsistencies. As a result, ClickHouse data can only be used for real-time data viewing, and Hive data will be used for final data use.
- Maintainability is complex. In business use, two sets of codes need to be developed to interface with the Hive and ClickHouse architectures.
- Query performance is not ideal. Among the three scenarios introduced above, it takes seconds or even minutes in ClickHouse for scenarios 1 and 2, and it takes more than ten minutes for scenario 3.
After introducing the OceanBase+Flink solution, data is written to OB in real time through Flink, and data cleaning is performed at the same time to standardize the data format. In Scenario 1 and Scenario 2, results can be returned in milliseconds. In Scene 3, the advertising effect can be seen in 1.5 seconds, and the performance improvement is very obvious.
The benefits of the new solution are also very obvious. Compared with the previous architecture, the performance ranges from minutes to seconds or even milliseconds, with fewer components and a lighter architecture. A set of solutions can meet the real-time requirements of some businesses, with low maintenance costs and low business transformation costs.
4. Future Outlook
In the actual implementation of the OceanBase and Flink solutions, we found that some optimizations can be made to Flink, mainly in the following three aspects:
- In terms of performance. Currently Flink is a single-threaded read data snapshot. In the future, snapshots will be cut into multiple data slices and read concurrently to improve performance.
- In terms of consistency. In the original design, in order to ensure that data is not lost, incremental reads are started first and then snapshot reads are started. When doing ETL operations, there may be data redundancy issues. In the new design, snapshot + incremental data reading can be optimized to achieve consistent reading.
- In terms of compatibility. Currently Flink is compatible with OceanBase connector version 5.1. As OceanBase is compatible with mysql 8.0, Flink will also need to adapt to the 8.0 connector in the future.
As OceanBase+Flink is widely used in the production environment, we will continue to adapt and improve the solution with Flink and surrounding ecological tools in the future to better serve enterprise users.