Quality is the lifeline of a product, and code inspection is a vital part of the software development process. It can help us discover and correct potential errors, improve software quality, and reduce maintenance costs.
This problem also exists in Kangaroo Cloud products . Due to the different SQL levels of offline data developers, code writing is confusing and there are many SQL code running problems. This article will introduce how to use SQL check rules to standardize SQL code in offline products , intercept code writing problems, facilitate unified management, and prevent the introduction of problems that need to be managed.
Through the introduction of this article, we hope that you can realize the importance of code inspection and understand how to improve code quality and development efficiency through best practices.
When should code rules checks be performed?
After the SQL task is developed on the offline product interface, if you click the run button, it will first be checked by the code rules. If the code rules are not met, the user will be prompted with specific reasons.
The data asset module has 5 built-in code inspection rules , which users can selectively turn on as needed.
After turning it on, you can select the code rule check items , effective scope and SQL task type to be used in offline project management .
Before running a SQL in an offline SQL task, the code will be checked according to the selected rules. If the code check fails, feedback will be given to the user. The user can judge whether to execute the SQL according to actual needs.
In the code inspection time of data assets, you can see the history of triggered inspections and corresponding statistics.
How to implement code inspection rules?
A common code rule checking interface is defined under the CodeCheck package.
public interface ICheck {
Result codeCheck(String checkContent, String defaultDb, Integer dataSourceType, Long tenantId, SqlParseInfo sqlParseInfo);
CodeCheckType getCheckType();
}
Taking the partition table query that must have partition rules as an example, the SQLParser component will be called first to perform SQL parsing. SQLParseInfo is the SQL parsing result. During the check, it will first determine whether the SQL statement is a query statement. If so, it will determine whether the queried table is a partitioned table. , then determine whether there is a query condition, and finally determine whether the query condition contains a partition field to determine whether the check passes.
public class CodeCheckImplType01 extends AbstractCheck {
private static final Logger LOGGER = LoggerFactory.getLogger(CodeCheckImplType01.class);
@Autowired
private DataTableColumnThirdService dataTableColumnThirdService;
@Autowired
private DataTableThirdService dataTableThirdService;
@Override
public Result codeCheck(String checkContent, String defaultDb, Integer dataSourceType, Long tenantId, SqlParseInfo sqlParseInfo) {
if (!isQuery(sqlParseInfo.getSqlType())) {
return Result.buildSuccessResult();
}
try {
MetadataSearchParam searchParam = new MetadataSearchParam();
searchParam.setDbName(sqlParseInfo.getMainTable().getDb());
searchParam.setTableName(sqlParseInfo.getMainTable().getName());
searchParam.setDataSourceType(dataSourceType);
searchParam.setTenantId(tenantId);
List<TableDTO> tableDTOS = dataTableThirdService.tableList(searchParam);
// 获取表信息
for (TableDTO tableDTO : tableDTOS) {
List<DataTableColumn> tableColumns = dataTableColumnThirdService.listColumnByTableIds(Lists.newArrayList(tableDTO.getTableId()));
if (CollectionUtils.isEmpty(tableColumns)) {
continue;
}
List<String> partitionColumnNameList = tableColumns.stream()
.filter(Objects::nonNull)
.filter(t -> HavePartitionEnum.have_partition.getPartitionValue().equals(t.getIsPartition()))
.map(DataTableColumn::getColumnName)
.collect(Collectors.toList());
// 非分区表直接返回
if (CollectionUtils.isEmpty(partitionColumnNameList)) {
continue;
}
if (CollectionUtils.isEmpty(sqlParseInfo.getColumnIdentifierList())) {
// 没有查询条件则校验失败
return Result.buildFailedResult(String.format(getCheckType().getCheckResultFormat(), searchParam.getTableName()));
}
List<String> columnList = sqlParseInfo.getColumnIdentifierList().stream()
.filter(c -> StringUtils.equals(c.getDb(), searchParam.getSchemaName()) && StringUtils.equals(c.getTable(), searchParam.getTableName()))
.map(ColumnIdentifier::getColumn).collect(Collectors.toList());
boolean disjoint = Collections.disjoint(partitionColumnNameList, columnList);
if (disjoint) {
return Result.buildFailedResult(String.format(getCheckType().getCheckResultFormat(), searchParam.getTableName()));
}
}
} catch (Exception e) {
// 异常情况先通过
LOGGER.error("code check error, check content: {}, defaultDb: {}, checkType: {}", checkContent, defaultDb, getCheckType().name());
}
return Result.buildSuccessResult();
}
@Override
public CodeCheckType getCheckType() {
return CodeCheckType.TYPE_01;
}
}
How to customize code inspection rules?
If the built-in code checking rules do not meet the customer's usage scenarios, customers can customize the code checking rules by uploading jars .
Custom code checking rules use the SPI mechanism to load custom jars uploaded by users, and call the CodeCheck method during code detection and the close method when resources are closed. Users need to depend on the jars in the configuration file description in their own projects. details as follows:
● Create a class that implements the interface
Create a class that implements the interface com.dtstack.assets.spi.codecheck.ICodeCheckClient and implements the CodeCheck and close methods, and writes relevant logic code. If the verification passes, set the success in the CheckResult object to true. If it fails, set the success field to false and Set the reason why the verification fails.
package com.dtstack.assets.spi.codecheck;
import java.util.Map;
public interface ICodeCheckClient {
/**
* 代码检查
*
* @param checkContent 检查内容
* @param extMap 扩展配置
* @return 检查结果
*/
CheckResult codeCheck(String checkContent, Map<String, Object> extMap);
/**
* 释放资源, 调用时需要关闭所使用的资源
*/
void close();
}
·Explanation of input parameter fields
– checkContent is a single piece of SQL information
– extMap will set some platform properties, including task name, task type, etc.
· Explanation of parameter fields
– success indicates whether the verification is passed and must be set
– checkResult is the verification result and cannot be empty when the verification fails.
package com.dtstack.demo;
import com.dtstack.assets.spi.codecheck.CheckResult;
import java.util.Map;
public class CodeCheckImpl implements com.dtstack.assets.spi.codecheck.ICodeCheckClient{
@Override
public CheckResult codeCheck(String checkContent, Map<String, Object> extMap) {
// 代码检查相关逻辑
CheckResult checkResult = new CheckResult();
checkResult.setSuccess(false);
checkResult.setCheckResult("校验不通过的理由");
return checkResult;
}
@Override
public void close() {
// 关闭相关资源
}
}
● Create the META-INF/services directory under the resource directory
● Create files in the META-INF/services directory
The file name is com.dtstack.assets.spi.codecheck.ICodeCheckClient, and the file content is the permission-defined class name of the class that implements the ICodeCheckClient interface.
Example of file name and content:
● Package the current project and register code verification rules on the data assets page
A prompt will be given for jars that do not meet the conditions.
How to load the jar corresponding to custom code rules?
We will initialize a unique custom classloader for the jar corresponding to each uploaded rule. This classloader inherits the URLClassLoader and ensures that the subclass loader loads first.
Load and cache the corresponding client on the first call.
After the user re-uploads or edits the rules, clear the old classloader and loaded client and release the resources.
"Dtstack Product White Paper": https://www.dtstack.com/resources/1004?src=szsm
"Data Governance Industry Practice White Paper" download address: https://www.dtstack.com/resources/1001?src=szsm Friends who want to know or consult more about Kangaroo Cloud big data products, industry solutions, and customer cases, please browse Kangaroo Cloud official website: https://www.dtstack.com/?src=szkyzg
At the same time, students who are interested in big data open source projects are welcome to join "Kangaroo Cloud Open Source Framework DingTalk Technology qun" to exchange the latest open source technology information, qun number: 30537511, project address: https://github.com/DTStack
Fined 200 yuan and more than 1 million yuan confiscated You Yuxi: The importance of high-quality Chinese documents Musk's hard-core migration server Solon for JDK 21, virtual threads are incredible! ! ! TCP congestion control saves the Internet Flutter for OpenHarmony is here The Linux kernel LTS period will be restored from 6 years to 2 years Go 1.22 will fix the for loop variable error Svelte built a "new wheel" - runes Google celebrates its 25th anniversary