What do the database logs commonly used in development look like?

Table of contents

Common log levels

database log

Common log levels

DEBUG: Used to record the running process of the application in detail, such as variable values, execution flow, etc. DEBUG level logs are usually used in the development and debugging process to understand the internal running status of the program.
INFO: Used to record general information about the application, such as state changes, operation results, etc. INFO level logs are usually used in production environments to understand the health of the application.
WARNING: Used to record events that may cause problems, such as performance bottlenecks, configuration errors, etc. WARNING level logs are usually used to alert developers to potential problems and risks.
ERROR: Used to record errors and exceptions that occur during the running of the application. ERROR-level logs usually contain information such as the time, location, detailed description, and stack trace of the error to facilitate troubleshooting.
CRITICAL: Used to record critical errors and problems in the application, such as data loss, system crashes, etc. CRITICAL level logs are usually used to deal with emergencies and major events.

These log levels are usually arranged in order from low to high, that is, DEBUG < INFO < WARNING < ERROR < CRITICAL. When configuring loggingthe library, you can set the minimum level of logging to facilitate filtering of unwanted log information.

database log

Let me talk about the three types of logs first (the main difference lies in the information recorded and the way to restore data):

Logical Log : The logical log records a high-level description of database operations , such as insert, update, and delete operations. Logical logs focus on changes in data rather than changes in underlying data structures. UndoLogIt is a logical log that records operations on database tables so that they can be undone when needed.
Physical Log : The physical log records changes in the underlying data structure of the database , such as changes in data pages on disk . The physical log focuses on the specific implementation of data storage, not on the logical changes of data. Physical logs are usually used in scenarios such as database failure recovery and backup.
The binary log (Binary Log, also known as Binlog) is a log file used to record database operations in the MySQL database. It records all change operations made to the database (such as inserts, updates, deletes, etc.) and when these operations occurred. Binary logs are stored in binary format . In a way it resembles a logical log in that it records a high-level description of operations.

Undo log logical log

Actually, UndoLogthe record format in the log file depends on your application requirements and logging style. Typically, records in a log file are stored as text, one record per line, which can be in JSON, CSV, or other custom formats. UndoLogThe following is an example of a log file record stored in JSON format :

{"id": 1, "operation": "UPDATE", "timestamp": "2022-08-01T12:34:56Z", "table_name": "User", "primary_key": 1, "old_data": {"name": "Alice", "age": 25}, "new_data": {"name": "Alice", "age": 26}}
{"id": 2, "operation": "INSERT", "timestamp": "2022-08-01T12:35:30Z", "table_name": "User", "primary_key": 2, "old_data": null, "new_data": {"name": "Bob", "age": 30}}
{"id": 3, "operation": "DELETE", "timestamp": "2022-08-01T12:36:10Z", "table_name": "User", "primary_key": 3, "old_data": {"name": "Charlie", "age": 22}, "new_data": null}

In the example, the log file contains 3 UndoLogrecords, which are an update operation, an insert operation and a delete operation. Each record is a JSON object, occupying one line. This format is easy to parse and process.

redolog binlog

Both redo log and binlog are stored in binary format and cannot be viewed directly. The
mysqlbinlogbinlog log records viewed by tools are similar:

# at 4
#211001 12:00:00 server id 1  end_log_pos 0 CRC32 0x12345678
Start: binlog v 4, server v 8.0.27-0ubuntu0.20.04.1-log created 211001 12:00:00 at startup

# at 123
#211001 12:00:00 server id 1  end_log_pos 0 CRC32 0x23456789
Query: thread_id=2 exec_time=0 error_code=0
SET TIMESTAMP=1633080000/*!*/;
BEGIN
/*!*/;
# at 456
#211001 12:00:00 server id 1  end_log_pos 0 CRC32 0x34567890
Query: thread_id=2 exec_time=0 error_code=0
SET TIMESTAMP=1633080000/*!*/;
UPDATE users SET age=26, last_login='2021-10-02 10:00:00' WHERE id=100
/*!*/;
# at 789
#211001 12:00:00 server id 1  end_log_pos 0 CRC32 0x45678901
Query: thread_id=2 exec_time=0 error_code=0
SET TIMESTAMP=1633080000/*!*/;
COMMIT
/*!*/;

Instead, redo logit is primarily used for internal database processing and usually does not require direct viewing or analysis. When data recovery is required, the database management system handles it automatically redo log.

slow query log

MySQL's slow query log records queries whose execution times exceed a specified threshold . Slow query logs can help you discover performance bottlenecks and optimize database performance. The format and content of the slow query log may vary depending on the MySQL version and configuration, but usually includes information such as the query's SQL statement, execution time, and lock time.

Simplified example

# Time: 211001 12:00:00
# User@Host: example_user[example_user] @ localhost []
# Thread_id: 1  Schema: example_db  Last_errno: 0  Killed: 0
# Query_time: 5.123456  Lock_time: 0.000123  Rows_sent: 1  Rows_examined: 1000  Rows_affected: 0
# Bytes_sent: 100  Tmp_tables: 0  Tmp_disk_tables: 0  Tmp_table_sizes: 0
# QC_Hit: No  Full_scan: Yes  Full_join: No  Tmp_table: No  Tmp_table_on_disk: No
# Filesort: Yes  Filesort_on_disk: No  Merge_passes: 0
SET timestamp=1633080000;
SELECT * FROM example_table WHERE some_column = 'some_value';

An operation was recorded SELECTinvolving example_tablethe table named. The query time ( ) for this operation Query_timewas 5.123456 seconds, which exceeded the slow query threshold. The slow query log also contains other metadata, such as lock time ( Lock_time), number of rows sent ( Rows_sent), query type (full table scan, file sort, etc.), etc.

To enable the slow query log and set the threshold, you need to add the following configuration in the MySQL configuration file (eg my.cnfor my.ini):

slow_query_log = 1
slow_query_log_file = /path/to/your/slow-query.log  # 日志文件路径
long_query_time = 1  # 慢查询阈值设置为1s

AOF text file

AOF (Append Only File) belongs to the persistent log of the Redis database. It records all commands to operate Redis so that data can be restored when the Redis server is restarted. AOF logs are stored in text format, and each line is a Redis command. Usually has .aof an extension such as appendonly.aof。

*2
$6
SELECT
$1
0
*3
$3
SET
$3
key
$5
value
*3
$5
LPUSH
$6
mylist
$4
item

The AOF file in the example contains three Redis commands: SELECT 0(select database 0), SET key value(set key-value pair) and LPUSH mylist item(push element into list). When the Redis server starts, it automatically executes these commands to restore the data.

RDB binaries

The Redis RDB (Redis Database) file is a binary file used to store the data snapshot of the Redis database at a certain point in time. RDB files typically have .rdban extension such as dump.rdb. Since RDB is in binary format, its contents cannot be read directly.

The RDB file contains the key-value pair data in the Redis database, as well as some metadata, such as expiration time, etc. The structure and format of RDB files are implemented internally by Redis and generally do not need to be concerned. (Close with a feint)