5. Error event handling
Errors should be considered in the design of the ETL system. Errors can be divided into two categories: Error and Exception.
Error level errors, the ETL system will stop running or skip a certain core module. This type of error generally requires an alarm mechanism, such as email or webhook alarm.
Exception level errors can be recorded in the log, and subsequent modifications or maintenance are considered. For example, the appearance and filtering of abnormal data values.
General ETL tools will integrate a complete error log function
Sample of error event alarm table:
| Error event |
|---|
| Error event id |
| Table Name |
| Timestamp |
| batch number |
| ETL module |
| severity |
| Error log |