Author: Yu Zhenxing

A member of the Acson DBA team, keen on technology sharing and writing technical documents.

Source of this article: original contribution

Produced by the Aikesheng open source community, the original content is not allowed to be used without authorization, please contact the editor and indicate the source for reprinting.

Background Information

In the process of data migration or upgrading from a low version of the database to a high version, it is often encountered that the time data inserted does not meet the specifications due to the loose setting of the database parameters of the low version, and an error is triggered. Trouble, is there a way to discover this kind of irregular data in advance? The following provides a feasible solution based on Oracle and MySQL as a reference.

Oracle time data verification method

2.1 Create a test table and insert test data

CREATE TABLE T1(ID NUMBER,CREATE_DATE VARCHAR2(20));

INSERT INTO T1 SELECT 1, '2007-01-01' FROM DUAL;
INSERT INTO T1 SELECT 2, '2007-99-01' FROM DUAL;            -- 异常数据
INSERT INTO T1 SELECT 3, '2007-12-31' FROM DUAL;
INSERT INTO T1 SELECT 4, '2007-12-99' FROM DUAL;            -- 异常数据
INSERT INTO T1 SELECT 5, '2005-12-29 03:-1:119' FROM DUAL;  -- 异常数据
INSERT INTO T1 SELECT 6, '2015-12-29 00:-1:49' FROM DUAL;   -- 异常数据

2.2 Create an error logging record for the table

Oracle can call DBMS_ERRLOG.CREATE_ERROR_LOGthe package to record SQL errors and record abnormal data, which is very useful.
The meaning of the parameters is as follows
- T1for the table name
- T1_ERRORTemporary table for logging errors in operations on the table
- DEMOis the user who owns the table

EXEC DBMS_ERRLOG.CREATE_ERROR_LOG('T1','T1_ERROR','DEMO');

2.3 Create and insert data into a temporary table to verify the validity of time data

-- 创建临时表做数据校验
CREATE TABLE T1_TMP(ID NUMBER,CREATE_DATE DATE);

-- 插入数据到临时表验证时间数据有效性（增加LOG ERRORS将错误信息输出到错误日志表）
INSERT INTO T1_TMP 
SELECT ID, TO_DATE(CREATE_DATE, 'YYYY-MM-DD HH24:MI:SS')
FROM T1 
LOG ERRORS INTO T1_ERROR REJECT LIMIT UNLIMITED;

2.4 Verification error record

SELECT * FROM DEMO.T1_ERROR;

The ID column is the primary key of the table, which can be used to quickly locate abnormal data rows.

Methods for MySQL databases

3.1 Create a test table to simulate low version non-standard data

-- 创建测试表
SQL> CREATE TABLE T_ORDER(
    ID BIGINT AUTO_INCREMENT PRIMARY KEY,
    ORDER_NAME VARCHAR(64),
    ORDER_TIME DATETIME);

-- 设置不严谨的SQL_MODE允许插入不规范的时间数据
SQL> SET SQL_MODE='STRICT_TRANS_TABLES,ALLOW_INVALID_DATES';

SQL> INSERT INTO T_ORDER(ORDER_NAME,ORDER_TIME) VALUES 
    	('MySQL','2022-01-01'),
    	('Oracle','2022-02-30'),
    	('Redis','9999-00-04'),
    	('MongoDB','0000-03-00');

-- 数据示例
SQL> SELECT * FROM T_ORDER;
+----+------------+---------------------+
| ID | ORDER_NAME | ORDER_TIME          |
+----+------------+---------------------+
|  1 | MySQL      | 2022-01-01 00:00:00 |
|  2 | Oracle     | 2022-02-30 00:00:00 |
|  3 | Redis      | 9999-00-04 00:00:00 |
|  4 | MongoDB    | 0000-03-00 00:00:00 |
+----+------------+---------------------+

3.2 Create a temporary table for data normative verification

-- 创建临时表，只包含主键ID和需要校验的时间字段
SQL> CREATE TABLE T_ORDER_CHECK(
    ID BIGINT AUTO_INCREMENT PRIMARY KEY,
    ORDER_TIME DATETIME);
  
-- 设置SQL_MODE为5.7或8.0高版本默认值
SQL> SET SQL_MODE='ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION';

-- 使用INSERT IGNORE语法插入数据到临时CHECK表，忽略插入过程中的错误
SQL> INSERT IGNORE INTO T_ORDER_CHECK(ID,ORDER_TIME) SELECT ID,ORDER_TIME FROM T_ORDER;

3.3 Data comparison

Do an associated query between the temporary table and the official table, and compare the inconsistent data.

SQL> SELECT 
	T.ID,
	T.ORDER_TIME AS ORDER_TIME,
	TC.ORDER_TIME AS ORDER_TIME_TMP
FROM T_ORDER T INNER JOIN T_ORDER_CHECK TC 
ON T.ID=TC.ID
WHERE T.ORDER_TIME<>TC.ORDER_TIME;

+----+---------------------+---------------------+
| ID | ORDER_TIME          | ORDER_TIME_TMP      |
+----+---------------------+---------------------+
|  2 | 2022-02-30 00:00:00 | 0000-00-00 00:00:00 |
|  3 | 9999-00-04 00:00:00 | 0000-00-00 00:00:00 |
|  4 | 0000-03-00 00:00:00 | 0000-00-00 00:00:00 |
+----+---------------------+---------------------+

a little trick

Use regular expressions to match the time field, and use the above method for rigorous requirements. Regular matching is brain-intensive.

-- Oracle 数据库
SELECT * FROM  T1 WHERE NOT REGEXP_LIKE(CREATE_DATE,'^((?:19|20)\d\d)-(0[1-9]|1[012])-(0[1-9]|[12][0-9]|3[01])$');

	ID CREATE_DATE
---------- --------------------
	 2 2007-99-01
	 4 2007-12-99
	 5 2005-12-29 03:-1:119
	 6 2015-12-29 00:-1:49
	 
-- MySQL 数据库
-- 略，匹配规则还在调试中

About SQLE

The SQLE of the Akson open source community is a SQL auditing tool for database users and managers, which supports multi-scenario auditing, supports standardized online processes, natively supports MySQL auditing, and has scalable database types.

SQLE get

type	address
Repository	https://github.com/actiontech/sqle
document	https://actiontech.github.io/sqle-docs/
release news	https://github.com/actiontech/sqle/releases
Data audit plug-in development documentation	https://actiontech.github.io/sqle-docs-cn/3.modules/3.7_auditplugin/auditplugin_development.html

How to verify MySQL&Oracle time field compliance?