SQL Audit | SQLE 2.2305.0 is here!

SQL Audit Tool SQLE 2.2305.0 was released today. The following is a detailed interpretation of the new version of Release Notes. The article is mainly divided into the following three parts:

1. SQLE project introduction
2. Introduction of the main functions of the new version
3. Complete Release information

1. SQLE project introduction

The SQLE of the Akson open source community is a SQL auditing tool for database users and managers, which supports multi-scenario auditing, supports standardized online processes, natively supports MySQL auditing, and has scalable database types.

SQLE get

type	address
Repository	https://github.com/actiontech/sqle
document	https://actiontech.github.io/sqle-docs-cn/
release news	https://github.com/actiontech/sqle/releases
Data audit plug-in development documentation	https://actiontech.github.io/sqle-docs-cn/3.modules/3.7_auditplugin/auditplugin_development.html
Community Edition Online Experience	http://demo.sqle.actionsky.com super administrator: admin password: admin
Enterprise Edition Online Experience	http://demo.sqle.actionsky.com:8889 User: admin Password: admin

2. Introduction of the main functions of the new version

[Community Edition]

1. Support to suspend SQL online operation

At present, in SQLE's SQL work order online process, the entire process of submitting SQL to online is well controlled, but the abnormal process after SQL online is not handled, and human intervention is still required in these extremely small scenarios, such as : When the online SQL is blocked, it needs to be manually killed; if the SQL execution fails, it needs to manually confirm the data impact, etc.

This version adds the function of suspending the SQL online. When the SQL online is blocked, the online operation can be suspended with one click on the work order interface, reducing the cost of manual intervention and reducing the recovery time of the failure. In the future, SQLE will also cover more abnormal scenarios, and the backup will be rolled back after the exception, and it will go online again after the exception. Create a perfect closed-loop process. The following is a simple function trial: the user enters the online work order, clicks the abort online button, and confirms the execution of the abort operation in the secondary confirmation pop-up window;

At this time, the status of the data source will change to fail to go online, the transaction that is executing the online operation will be rolled back, and the transaction that has already been executed will not be rolled back. Users can click on the data source to view the execution status of the specific SQL;

2. Support notification of work order status through Webhook

When the user has configured a custom API service and wants to obtain the execution status of the work order in the API service in real time, the Webhook service can be configured in the system settings. After the callback address is configured in SQLE, once the work order status changes, the status notification information can be obtained in the API service, and there is no need to log in to the SQLE platform regularly to query the results. The following is a simple function trial: 1. Log in with the platform administrator account, enter the system settings, find the Webhook configuration and click the Modify button.

Select Enable Webhook Notifications and fill in the following fields.

Webhook url: the API address used to push ticket messages;
Maximum number of words to retry: 3 times by default, value range 0-5. If the event push fails, SQLE will push the notification again. If the push fails after 3 retries, the event will not be pushed again;
Maximum retry interval: default 1 second, value range 1-5. If the push fails, it will be re-pushed at an interval of 1s;
token: used to identify the pusher, such as SQLE can be filled in to identify the notification pushed from SQLE.

Click Submit to save the current webhook configuration.
Users can click Test to verify that the API server can receive push information normally.

[Enterprise Edition]

1. Smart scan supports online crawling of slow SQL from MySQL instances

In the original slow log scanning task, after creating the scanning task, you need to enter the terminal configuration scannerd process, and the scanning task configuration process is relatively long. This optimization enables sqle to directly collect data from the slow log table, avoiding users from entering the terminal operation, and reducing the process obstacles when users use the slow log scanning task.

The following is a simple user experience:

Create a new scan task, select slow log scan, and set the collection source to 1, and directly query the table of the instance through SQLE . mysql.slow_log

New parameter: collection source
- If the source is set to 0: the scan task will be collected by scannerd mysql-slow.log
- If the source is set to 1: the scan task will directly query the table of the instance through SQLE mysql.slow_log

To configure a database instance, you need to pay attention to the following settings.

sql

set global long_query_time=1; // 需根据实际业务调整慢查询时间阈值，此处仅供参考 set global slow_query_log=1; // 开启慢日志查询 set global log_output='FILE,TABLE'；// 开启 MySQL 的慢日志收集功能，并且确认将慢日志内容写入文件 mysql-slow.log 和表 mysql.slow_log

Execute a slow SQL in the database, click Audit Now, and you can view the statistics of the slow log scan.

2. DB2 database table metadata scan task

If the table metadata in the database has no comment on the table, no comment on the column name, unreasonable index in the table structure, lack of primary key, etc., if it is not processed in time, it will eventually lead to low performance of some business requests and loss of actual business traffic. For this scenario, users can use the database table metadata scanning task to perform real-time intelligent scanning of the database table metadata of the instance, helping users to discover irregularities and unreasonable database table metadata in advance, and provide optimization suggestions. Here is a simple function trial:

Create a scan task. After the scan task is established, SQLE will store the query results and review them according to the period set by the user.

Users can view the SQL collected by the current scan task in the scan task details, and click " Approve Now " to generate a scan task report.

3. DB2 Top SQL scan task

TOP SQL statements may cause database performance degradation. When users need to monitor the most resource-intensive SQL statements in the DB2 database and perform regular optimization to improve database performance, they can use the TOP SQL scanning task for daily monitoring to ensure database performance. at a good level. Currently collected SQL supports four indicators for query sorting, namely:

Number of executions num_executions
Total execution time total_elapsed_time_ms
Average execution time avg_elapsed_time_ms
Average CPU time avg_cpu_time_ms

The following is a simple function trial: 1. Create a scan task. After the scan task is created, SQLE will store the query results and review them according to the period set by the user.

Users can view the SQL collected by the current scan task in the scan task details, and click " Approve Now " to generate a scan task report.

4. Support DB2 more audit rules

The platform strengthens the development of DB2 audit rules. Currently supported DB2 audit rule categories include DDL specifications, DML specifications, DQL specifications, usage suggestions, naming specifications, and index specifications. Users can enable rules according to actual needs to optimize DB2 data SQL on source.

5. Oracle supports generating rollback statements

When the user performs SQL online operation on the Oracle data source through the platform, if the work order fails to go online, it may cause only part of the SQL in the work order to be executed but not other parts. For SQL that has already been executed, the user may not know how to roll back the operation. To this end, SQLE provides a rollback statement hint to help users roll back the operations that have been performed to ensure data consistency and integrity. The following is the prompt effect of the rollback statement under the Oracle data source:

6. PG supports generating rollback statements

In this issue, the rollback statement support for the PostgreSQL data source has been added in the review process, which is convenient for DBAs to perform offline rollback operations. Here is a brief example:

7. PG's SQL analysis supports viewing table creation statements

This issue improves the SQL analysis support under the PostgreSQL data source. In addition to the original column information and index information, the display of table creation statements is added. Users can view the complete table creation statements here to help DBAs optimize SQL. Here is a brief example:

The user clicks the analysis button in the review result column;

Click the table label to view the table creation statement of the SQL execution object;

3. Complete Release information

New features:

[#1519] Support aborting SQL online operation
[#450] Support ticket status notification via Webhook
[#1494] Add DB2 TOP SQL and database table metadata smart scan type【Enterprise Edition】
[#1461] Intelligent scan supports online crawling of slow SQL from MySQL instances【Enterprise Edition】
[#715] Oracle and PostgreSQL support rollback statements【Enterprise Edition】
[#706] Support PostgreSQL data source to generate rollback statements【Enterprise Edition】
[#714] Support SQL analysis of PostgreSQL data source to view table creation statement【Enterprise Edition】

optimization:

[#1508] UI optimization
[#1473] Automatically switch the default port according to the data source type when adding a data source

Bug fixes:

[#1480] Fix the rejected work order, the operation button is grayed out, but it can actually be clicked
[#1495] Fix the problem that the MySQL audit rule "avoid using IN (NULL)/NOT IN (NULL), which is never true will cause the condition to fail" is falsely triggered
[#1499] Fix the problem that the audit result of the work order does not display the SQL syntax highlighting
[#1399] Fix the problem that the index has capital letters, and the audit error is reported when creating a work order
[#1463] Fix a problem that a MySQL rule cannot be triggered
[#1360] Fix the problem that the progress of the work order is displayed abnormally after modifying the process template