DTCC 2020 | Alibaba Cloud Chengshi: Database Management in the Cloud Native Era

Introduction: With the continuous development of cloud native technology, databases have gradually entered the cloud native era. In the cloud-native era, how to efficiently, securely and stably manage the databases on and off the cloud has become a major problem for enterprises. At the 11th China Database Technology Conference (DTCC2020), Alibaba database ecological tool team senior technical expert Cheng Shi (name: Shiqin) shared the database management system and solutions in the cloud native era for everyone.

Abstract: With the continuous development of cloud native technologies, databases have gradually entered the cloud native era. In the cloud-native era, how to efficiently, securely and stably manage the databases on and off the cloud has become a major problem for enterprises. At the 11th China Database Technology Conference (DTCC2020), Alibaba database ecological tool team senior technical expert Cheng Shi (name: Shiqin) shared the database management system and solutions in the cloud native era for everyone.

The content of this article is organized based on speech recordings and PPT.

This sharing will introduce how to combine and use Alibaba cloud native database management system products to create a complete solution for users.

Cloud native database management system

The main characteristics of the cloud-native era can be roughly summarized into two points, namely, hardware characteristics and software characteristics. The former refers to the high availability and elasticity brought by hardware resource pooling; the latter refers to the cloud-native era, many tools There is no need for the company to research and develop itself, but the combination of software functions can be called through API, and then organically combined into a software ecosystem.

So, what technologies are needed for database management in the cloud-native era? In fact, it can be divided into three main categories, namely management, migration and backup. Among them, management requires security, intelligence, and capabilities for hybrid clouds; the main challenge for migration lies in the heterogeneity of databases; and for backup, what needs to be solved today is not whether the backup is successful, but how to activate the value of the data .

Alibaba Cloud database tool system

The above figure shows the database tool system of Alibaba Cloud. As can be seen from the figure, first of all, tools such as DBS, DTS data migration, and database backup can be used to connect databases from other clouds or clouds to Alibaba Cloud, and it can also make the database on Alibaba Cloud and the database on Alibaba Cloud The computing platform, streaming computing platform and data analysis platform are connected. Second, through database management services such as DMS, computing and analytical databases on and off the cloud can be managed. Third, Alibaba Cloud also provides the database autonomous service DAS, which is an intelligent autonomous platform for optimizing database management related services.

How can non-cloud databases embrace cloud native? ——Cloud Native Empowerment

Next, we will analyze a common user problem. For users, if a large number of existing databases are non-cloud database instances, how should they embrace cloud native technology? We can certainly introduce a cloud native database product for it, but what users want to know is how to use this product, how to migrate data, and how offline database instances work with cloud native databases. In other words, for users, what is actually needed is a set of solutions.

Alibaba Cloud Database provides users with three solutions, which can be applied to different scenarios.

Option 1 is to migrate to a cloud database as a whole. In this kind of scheme, users are most concerned about the compatibility of database and application. In many cases, users need to migrate from an Oracle database to a cloud database or from their original historical version database, so they need to be concerned about compatibility issues. In addition, for this kind of solution, the transmission performance also needs to be concerned.
Option 2 is that the data is still under the cloud, but cloud-native technology should be used to manage the data under the cloud, that is, "data under the cloud, management on the cloud." For this solution, it is equivalent to running a database management software on the cloud, which can help users save various labor costs of database operation and maintenance. For users, what they need to care about is how to manage services and services on the cloud. Establish a secure channel between data under the cloud. Second, we need to pay attention to data security and efficiency issues. Finally, we need to consider the issue of intelligence in depth, because this solution is not to move people's abilities to the cloud, and to turn offline manual operations into online operations, but rather intelligent management on the cloud. , Which will greatly reduce the burden on DBA and operation and maintenance personnel.
Option 3 is that the production data is under the cloud, and the secondary data is on the cloud. How to understand the relationship between production data and secondary data? In fact, production data is the data generated by the business. For example, the data generated by Taobao's exchange is directly written into Alibaba's buyer database or production transaction database. Such data is called production data. The secondary data is the backup of the production data. The production data is copied to the Hadoop platform, the Elastic platform and other places where the data is backed up. These data are all called the secondary data. In Gartner's recent report, it is mentioned that in the current Internet industry, the ratio of secondary data to production data has reached 20:1, which means that production data only accounts for a small part of all management data, while secondary data is instead. Becoming a big head may even be the main source of consuming assets.

Next, introduce some database tools provided by Alibaba Cloud.

DMS data management-best practices for rights management

There is the concept of "separation of powers" in DMS data management products, which is also a best practice for database management in large enterprises. Here is a general introduction. First, there must be an administrator or super administrator role, usually performed by an operation and maintenance leader or CTO, who is mainly responsible for defining personnel roles and assigning personnel permissions. Secondly, there is usually a DBA or operation and maintenance personnel who is responsible for the actual operation and maintenance of the database, such as capacity expansion and schema changes. The last one is also more concerned now, called the security administrator, who is responsible for formulating data operation specifications, such as how many times a day can be operated, in what window can this operation be completed, and the scope of an operation. On the one hand, the security administrator restricts the DBA, and on the other hand restricts the fourth role, that is, ordinary users. For example, business users only need to read and write a certain database, but do not need to operate the database instance, they need to be subject to the security administrator’s Constraints; For example, in some high-risk businesses or highly sensitive businesses, business personnel do not want operation and maintenance personnel to see specific data, especially for cloud services, so they also need to be constrained by security administrators. The security administrator can restrict each type of ordinary user on the DMS. He can see or even restrict the data that the DBA can see, and can also define data desensitization processing to achieve row-level data query constraints.

DMS data management-to get through with the company's internal account SSO system

The biggest obstacle to the implementation of the above-mentioned data management best practices is that the management of people and permissions is not in place. For a company, there must be a liquidity process such as personnel entry, resignation, and department replacement. Therefore, a strong correlation between personnel permissions and their current locations must be realized, so that effective management is possible, otherwise the permissions system will collapse. DMS can support the connection with the enterprise's single sign-on system SSO by providing the OpenAPI system, that is, the enterprise's authority management system. Regardless of whether it is employee entry, department change, position or position adjustment, the permissions can be changed by linking with the DMS OpenAPI in the system, so that employee permissions are always consistent with their positions and departments.

DMS data management-change security / R & D efficiency

There are many more functions provided by Alibaba Cloud DMS, such as a change order system for research and development performance, cross-database query, etc., SQL auditing for change security, data desensitization, log tracking and rollback, and table changes without locking. Among them, DMS can provide intelligent analysis capabilities in terms of data desensitization. Even if you do not tell DMS which data is sensitive data, it can also intelligently analyze it; in terms of log tracking and rollback, DMS can analyze a row of data from the log Record the history of modification, not just show the final data modification results, and can help users automatically generate rollback SQL. In terms of unlocking table changes, it is equivalent to changing the Schema or bulk DML changes submitted by the user into a data copy, making changes on the copied copy, and then completing the replacement operation.

DAS database autonomy-automatically optimize parameters

DAS is a database autonomous product. Here we focus on two VLDB papers published by DAS last year and this year. In 2019, he published a paper entitled "iBTune: Individualized Buffer Tuning for Largescale Cloud Databases", which mainly uses machine learning technology to intelligently adjust the cache size. When the DBA only manages less than 10 databases, the brain can be used to memorize information such as the cache size, cache hit rate, and business RT requirements of each database. For a situation like Alibaba, a small number of DBAs are needed to deal with it. Million database. In addition. It is also very difficult to manually adjust the cache when the user does not have the knowledge of database tuning. The above paper attempts to solve such a problem. It uses machine learning methods to predict how the cache will be reduced, what will happen to the cache hit rate and the value of RT, which uses deep neural network technology. The benefit of using such prediction technology in tens of thousands of database instances within Alibaba is to reduce database memory usage by 12.44%.

DAS database autonomy-automatically optimize SQL

The picture above shows the paper titled "Diagnosing Root Causes of Intermittent Slow Queries in Cloud Databases" published by Alibaba DAS on VLDB this year. The main thing it does is to intelligently analyze slow queries, and the operations after analysis are automatic SQL Perform current limiting, automatic optimization of SQL, and automatic expansion. Such a thing may seem to be something that the DBA can decide with his head, but when faced with hundreds of databases, it is very difficult to understand which SQL each database has, and which SQL causes which problems. It is impossible for humans to do it, and machine learning methods must be used.

DAS database autonomy-self-repair, self-security, self-operation and maintenance

As far as database autonomy is concerned, when to limit the current and when to expand, these two scenarios are actually different. If based on the original traffic forecast, the traffic will fall after this period of time, then you should not expand the capacity at this time, because this is a short-term peak traffic, and it is not appropriate to expand the capacity for a short burst of peak traffic. It can be current limited at this time. But if the traffic continues to rise and it is predicted that the traffic will become higher and higher in the future, then the capacity should be expanded. DAS can automatically complete the forecast without the need for manual decision-making by DBA. In addition, DAS also provides a series of functions such as anomaly detection, fault detection, SQL audit, anti-injection, and security vulnerability repair. For details, please refer to the official DAS document.

DG database gateway-open up private network, local IDC and other cloud

The main function of DG is to connect online cloud database and offline. One of the solutions mentioned above is to manage online and offline user data. Users can easily feel insecure and may think that a public network port of a database needs to be opened offline. Easy to manage. But the solution provided by Alibaba Cloud is not like this. It does not need to expose any external ports. By establishing a reverse TCP channel, one end of the DG is connected to the offline database, and the other end is connected to the online management service, such as DMS and The DAS service completes the management of the database, and its channel is encrypted.

DTS data transmission-overall architecture

In fact, Alibaba was the first to provide data transmission services. Before the birth of Alibaba Cloud, DRC, the predecessor of DTS, was used for data transmission within Alibaba. The technologies that Alibaba was proud of in the early days, such as living in different places, were all point-to-point. Data synchronization is complete. As a cloud service, the core competitiveness of the current DTS is to support a lot of source and destination database types, and the realization of this is not simple, because the log structure and distributed architecture of different databases are different, so I want to support these Databases often require a lot of exploration, so there is a very high technical threshold. In addition, in addition to supporting migration from database to database, DTS also supports migration to some analysis platforms, or migration to some streaming computing platforms through subscription, and the subscription interface is also compatible with Kafka's SDK.

DTS data transmission-core technology: transaction-level real-time synchronization

DTS has concurrent synchronization at the transaction level. In fact, Alibaba Cloud DTS was the first to study transaction-level real-time synchronization, even earlier than the official MySQL. In addition, double writing needs to be prevented under high availability conditions, loops need to be prevented under multiple active states, and errors need to be addressed during data playback.

ADMA professional migration-the overall process

ADAM is an expert data migration service. The reason why the adjective "expert" is added in front is because it not only performs heterogeneous migration of data and solves the storage problem of data at both ends, but also helps users perform Database selection. ADAM will try to analyze the database that users use offline. For example, ADAM can help users analyze SQL, stored procedures, etc., and can analyze whether a database supports these online. If not, what other solutions can be used. This is also a database. The problem that the architect or DBA needs to solve, and the selection of the target database has been well solved in ADAM. Secondly, ADAM can realize compatibility analysis and automatic SQL transformation. This is because even the written SQL is different between the database and the database, not to mention the query SQL and Join query, although they all follow the SQL 99 specification , But all have SQL grammars in their respective dialects. Finally, there is automatic data correction, which can even guide the application to modify the application configuration and parameters to adapt to the new database, so it is called expert service. ADAM is about to launch a data migration laboratory, users will be able to try data migration without actually moving the data to analyze the difference between SQL and database management.

ADAM-core technology: SQL syntax conversion

The above figure shows the technical diagram of ADAM on SQL syntax conversion. ADAM converts the source SQL into target SQL by marking the AST syntax tree, and outputs a conversion report for problems that occur during the conversion.

DBS database backup-overall architecture

DBS database backup is a product that Alibaba Cloud is currently focusing on. The above figure shows the overall technical architecture.

DBS database backup-core technology: physical full PIT copy, physical incremental CBT

Simply share the core technology used in DBS backup. The first is physical backup technology, which allows users to complete data backup at high speed without perception. At present, the industry has to face two main technical problems in physical backup. One is the PIT copy (Point-in-time Copy) problem of full backup. Faced with this problem, the industry has proposed five technical solutions, namely Copy with Lock , Copy with Log, Copy on Write, Copy on Redirect and Mirror Split. The Copy with Lock solution is the simplest. It is to add a lock. For example, if you copy a MySQL database, you can lock all of it and copy the files, so that the copied files must be consistent. The so-called Copy with Log is to copy the log while copying the data, and then replay the copied log during this period. MySQL uses Xtrabackup, and Oracle uses RMAN, which are all based on this scheme. The VSS mechanism that the SQL Server database relies on is the Copy on Write scheme. There is no log, but there is a layer of agreement in the business and storage layer interface to achieve consistent copy. The Mirror Split solution is mainly used for special equipment. The second problem is the incremental backup CBT (Changed Block Tracking). In the incremental backup, it is necessary to discover which data blocks have been modified from the last time to this time. This discovery capability is used for different databases. The technology is also different, for example, Oracle uses RMAN and SBT. Windows uses VSS, and for MySQL, the official Percona’s XtraDB provides the CBT function. For MySQL that is not this way, the changed data blocks must be found by scanning. Therefore, it is expensive to use Xtrabackup to back up data. Alibaba Cloud DBS has done a lot of optimization work in these aspects.

DBS database backup-core technology: data lake analysis

As mentioned earlier, I hope to put the secondary data on the cloud and play the value of the data. The above figure shows that after the data is backed up to the object storage, a data lake is directly built, and then the data is submitted to the SQL query analysis, which means The data does not need to be restored to a certain database. You can directly perform SQL queries on the backup data, which is very user-friendly. For example, the query of historical data can be completed only on the backup data.

DBS database backup-core technology: CDM sandbox

The aforementioned solution actually involves multiple calls to the object storage interface, so its performance is not as good as that of a native database, but Alibaba Cloud provides a better solution. If you do a physical backup, you can use physical backup technology plus storage virtualization technology, plus cloud native technology to help users directly create a new instance online in seconds, and there is no limit to the number of creations. This technology is called Copy Data Management in the industry, and it has recently become popular in the professional field of data backup. Alibaba Cloud is able to pull up a new sandbox instance in seconds at any point in time. This instance can be used as an emergency disaster recovery object or used for development testing.

Finally, corresponding solutions are given for the three problems mentioned above.

Solution 1: DTS+ADAM-overall migration to the cloud

The first solution is to go to the cloud as a whole. Alibaba Cloud gives the technology combination solution of ADAM+DTS. ADAM completes database profile analysis and cloud database selection, generates a migration plan and performs structural migration and correction. DTS performs full and incremental migration of data. Finally, ADAM also conducts profile evaluation for offline applications. The main problem that the first solution needs to solve is the database heterogeneity problem and open up heterogeneous channels.

Solution 2: DMS+DAS+DG——Data under the cloud + management on the cloud

The second solution is for the situation of data under the cloud and management on the cloud. First, the secure network is connected through DG, then the offline database is managed through DMS and the offline database is intelligently operated and maintained through DAS. Database users with different identities face different platforms to use different data services.

Solution 3: DBS+DMS+DAS: production data under the cloud + secondary data on the cloud

The third solution is production data under the cloud + secondary data on the cloud. First, the data is backed up to the cloud through DBS, and the sandbox instance is generated through CDM technology, which can not only meet the needs of emergency reading and writing, but also provide for development and testing in seconds. Database copy, and can realize the operation and maintenance management of database sandbox instance through DAS.

Click here to download the presentation PPT