Introduction: Faced with pain points such as diversified database types, different functions, and low cross-client management and control efficiency, how to choose a tool that can not only support unified access points, but also ensure the security of data sources and achieve efficient data operations? ? As a practitioner, a large domestic insurance company will share its choices from four aspects: business background and pain points, product selection and product customization, problems encountered in using CloudQuery (herein referred to as "CQ"), and future plans for cooperation with CQ. process and thinking.
About the author: Wang Ke currently works for a large insurance management company as a DBA. He is mainly responsible for PolarDB localized database, open source database, database security management and control platform projects, and database knowledge operation and maintenance related projects.
Business background and pain points
In the current technological environment, we face significant challenges in data management and processing. With the rapid development of technology, various diversified database products have emerged. They are not only of many types, but also of different forms. There are more than a thousand domestic databases, more than a hundred open source databases, and some traditional databases such as Oracle and MySQL. database. These database products have their own advantages and characteristics and can meet different business needs.
However, for our insurance industry, technological advancement has also brought about two very difficult problems:
1. Lack of a unified database operation client
2. Permission control is not easy to control, and sensitive operation monitoring and security audits are relatively lacking.
Lack of a unified database operation client
Currently, there are many types of domestically produced database clients with different functions. For example: PolarDB will use Polar Stack, OceanBase will use OCP, and Dameng will have its own management tool interface. These databases involve more copyrights.
At the same time, the efficiency of cross-client management and control is not high. Taking the PolarDB database I am currently managing as an example, more than a hundred databases are distributed on more than ten Polar Stack clusters. For each database operation, the IP and port for database access must be entered on the corresponding Stack cluster server. This means that if I want to operate and maintain 10 sets of databases, I have to log in to 10 IPs and ports, and I have to span different server clusters. Such operation and control efficiency is not very high.
For open source databases, most of the current open source databases use third-party controlled clients. It is difficult for such third-party controlled clients to meet the security control standards stipulated by the company.
Permission control
Permission control is mainly reflected in the monitoring of sensitive operations and the relative lack of security audit content.
Faced with the above two relatively difficult problems, we need to build a unified security access port, allocate permissions, achieve efficient data operations, and conduct security audits for the entire process. Let’s talk to you about the problems that need to be solved for different needs.
- Unified security access >> All data is accessed through a unified platform, standardizing the way internal personnel connect to the database and facilitating subsequent centralized management and control.
- Permission allocation >> Determine the user's functions and application scenarios, and allocate initial fine-grained permissions to users based on the importance of database resources to avoid the risk of violations and information leakage caused by high-privilege abuse.
- Data Operation >> It is hoped that the built-in functions can fit industry application scenarios, improve operation and maintenance efficiency, and provide functions such as data desensitization and data export.
- Behavioral audit >> Hope it is a full-cycle audit with wide coverage. Based on the audit results report, you can get an overview of data execution and security status, helping internal personnel understand the overall execution security situation.
Product selection and product customization
Based on the above pain points and needs, we need to choose a database security management and control platform. After doing some market research, we finally focused on two products.
In order to accept more database types, especially those that can meet the standards of customized domestic database types, we need to provide stable services to support the problems we may encounter when facing new databases. Based on these needs, we finally chose CloudQuery and used CloudQuery 2.3 Enterprise Edition as the first project cycle in 2021-2022.
The database types involved in this project include: Ocenbase, PolarDB, Dameng, mongodb, mysql, redis, postgreSQL, Oracle. Deployed in a cluster manner, it is estimated that there are more than a thousand users within the company, and the number of connected databases has reached 1,000+.
After describing our needs with CloudQuery, CloudQuery also provided corresponding solutions. For the lack of a unified client, CloudQuery's solution is a self-developed unified database operation client built into the platform, which can accept a variety of databases . .
For the problem that database operations are difficult to control and permissions cannot be monitored, and sensitive behaviors cannot be monitored, unified database users and permissions management are implemented, and unified permissions control, security auditing, and behavior monitoring and alarming are implemented for database operations.
In view of the global control of permissions, the standard of unique user access, and the localization needs of the underlying database currently required by ten companies, we have also customized some content such as global permissions view, user center docking, and replacement of the platform's underlying database with PolarDB, behavior monitoring alarms, and cross-schema data export in bulk... This set of content is currently online and applied in multiple departments of the company.
Generally speaking, the launch of this set of customized products meets the needs of domestic digital unified management and control. At the same time, unified management of large batches of databases is achieved, improving management, control and operation efficiency.
The picture below is a detailed introduction to how the customized products mentioned above are used in our company:
Problems encountered when using CQ
Our project cycle is concentrated in 2022. During this project, we also encountered some usage problems. I simply summarized the problems into three major categories:
1) The problem of adapting various database drivers in the early stage of the project, especially the problem of adapting localized database drivers like OBPD.
2) Functional issues, such as CQ's support for special functions built into different databases, the design of global permission views, batch export functions, and the continuous improvement of audit pages, such as increasing the monitoring of SQL execution and high-frequency user monitoring.
3) Performance issues, such as the execution efficiency of OceanBase and MongoDB, the impact on the stability of the workbench cluster, and the replacement of OceanBase streaming database to improve the user experience.
Although there will be some problems in the initial stage of launch, a new product also needs to be constantly adapted to our production environment. Only by optimizing the product through the problems encountered in practical applications can we discover and build a product that suits us. During the entire project, CQ also gave us great support, which ultimately enabled CQ to better provide stable and diversified database management and control support.
Talking about CQ 2.4 user experience
In early July, CQ 2.4 was officially released, and I tried it out for the first time. Here I will talk about my experience. Generally speaking, CQ 2.4 has richer functions and is more in line with the needs of our production business in many aspects. , the following simple example:
- From the perspective of supported database types, 2.4 has completed the adaptation to GaussDB, which is what we expect when using version 2.3.
- From the perspective of database operation permissions, CQ has added a schema filtering function in the "Resource Management" module. This allows security auditors to filter the schema that comes with the database and is useless for business user authorization before authorization. This is clearer and more concise for authorization.
- Authorization management has also undergone major changes. CQ 2.4 divides various databases into small categories. Each category of databases will have a subset of N database links, so that the custom permissions of this category can be uniformly defined and authorized.
As can be seen from the picture above, the permissions are more detailed, and the unique permission standards can be customized according to your own needs. - From the perspective of security management and control functions, version 2.4 pays more attention to security management and control. As can be seen from the figure below, data desensitization, data protection, and auditing have all been set up as independent functional modules, and the functions under each module are richer and more detailed, and can better meet the standards of security control. It also supports text import, editable and copyable result sets, backup of the database from the original database to the target database, and display of monitoring associated permissions for audit details.
future plan
In the second half of the year, we will gradually complete the replacement of the old and new versions. After the version is stable, we will also work with CQ to formulate a data migration plan. The migration of part of the production environment will be completed first. When the production environment is stable, all database links on 2.3 will be replaced and officially switched to 2.4.
In addition, we will continue to improve functional requirements and fix new problems during use. We also hope to improve the progress and efficiency of the project through cooperation with CQ. Finally, we also hope that CQ can develop better as a product. , to provide strong support for more users.