Article directory
Preface
The so-called slow SQL refers to statements whose execution time in the database exceeds the specified threshold. Too many slow queries are very risky for the business and may be triggered at any time for some reason. According to our experience, the most common problems in the database are caused by slow queries that cause the database to be slow. , which in turn causes the entire instance to "avalanche" and lead to online failures. This article will introduce how to locate and troubleshoot PostgreSQL slow SQL.
1. Log parameter settings
MySQL provides users with slow_query_logparameters to set slow log storage:
# 是否开启慢日志
slow_query_log = 1
# 慢日志文件路径
slow_query_log_file = '/logs/slow.log'
# 慢日志阈值，取值范围 [0.000000-3600.000000]
long_query_time = 1
PostgreSQL also provides related parameters:
# -1 表示不记录语句
log_min_duration_statement = 100ms
# none, ddl, mod, all
# none: 表示不记录慢 SQL
# ddl: ddl 记录所有数据定义语句，例如 CREATE、ALTER 和 DROP 语句
# mod: DDL + INSERT, UPDATE、DELETE、TRUNCATE, 和 COPY FROM
# all: 所有语句都会被记录
log_statement = 'mod'
2023-09-14 14:07:46.695 CST [46385] LOG: statement: update pgbench_accounts set bid = 11 where abalance = 101;
2023-09-14 14:07:53.698 CST [46385] LOG: duration: 7003.518 ms
This is putting slow SQL and error logs into one file. I personally don’t think it looks good, especially when there is a lot of content. If anyone has other good methods or experience, you can post it in the comment area.
2. pg_stat_statements plug-in
The pg_stat_statements module provides a means of tracking all SQL statements executed by the execution statistics server. This module is not enabled by default. If we need to enable it, we need to manually compile and install it, modify the configuration file and restart the database, and manually load the module before use.
2.1 Confirm whether to install the plug-in
Before installing the plug-in, you must first confirm whether the plug-in has been compiled. You can check it in the PostgreSQL installation directory:
ll ./share/extension
Output:
总用量 52
-rw-r--r-- 1 root root 1246 9月 14 15:38 pg_stat_statements--1.0--1.1.sql
-rw-r--r-- 1 root root 1336 9月 14 15:38 pg_stat_statements--1.1--1.2.sql
-rw-r--r-- 1 root root 1454 9月 14 15:38 pg_stat_statements--1.2--1.3.sql
-rw-r--r-- 1 root root 345 9月 14 15:38 pg_stat_statements--1.3--1.4.sql
-rw-r--r-- 1 root root 305 9月 14 15:38 pg_stat_statements--1.4--1.5.sql
-rw-r--r-- 1 root root 1427 9月 14 15:38 pg_stat_statements--1.4.sql
-rw-r--r-- 1 root root 376 9月 14 15:38 pg_stat_statements--1.5--1.6.sql
-rw-r--r-- 1 root root 806 9月 14 15:38 pg_stat_statements--1.6--1.7.sql
-rw-r--r-- 1 root root 191 9月 14 15:38 pg_stat_statements.control
-rw-r--r-- 1 root root 449 9月 14 15:38 pg_stat_statements--unpackaged--1.0.sql
-rw-r--r-- 1 root root 310 9月 5 10:21 plpgsql--1.0.sql
-rw-r--r-- 1 root root 179 9月 5 10:21 plpgsql.control
-rw-r--r-- 1 root root 370 9月 5 10:21 plpgsql--unpackaged--1.0.sql
You can see some information about the pg_stat_statements file, indicating that the plug-in has been installed. If you see relevant files, it means that the plug-in has not been compiled and needs to be compiled before it can be used.
2.2 Compile plug-in
Enter the PostgreSQL source code directory and execute the following command:
# 进入插件目录
cd contrib/pg_stat_statements/
# 编译安装
make && make install
2.3 Load plug-in
After confirming that the compilation and installation are successful, you can use the plug-in. First, you need to modify postgresql.confthe configuration file:
# 加载动态库
shared_preload_libraries='pg_stat_statements'
# 记录语句的最大行数，默认 5000
pg_stat_statements.max = 10000
# 控制哪些语句会被该模块计数。指定 top 可以跟踪顶层语句（那些直接由客户端发出的语句）
# 指定 all 还可以跟踪嵌套的语句（例如在函数中调用的语句）指定 none 可以禁用语句统计信息收集。
pg_stat_statements.track = all
# 控制模块 是否 跟踪除了 “增删改查” 之外的命令，默认为 on
pg_stat_statements.track_utility = on
# 指定是否在服务器关闭之后还保存语句统计信息，默认为 on 关机前会持久化保存
pg_stat_statements.save = on
Then 重启数据库it will take effect.
Then use psql to connect to the PostgreSQL service and load the plug-in:
postgres=# create extension pg_stat_statements;
CREATE EXTENSION
Check plug-in status:
postgres=# SELECT * FROM pg_available_extensions WHERE name = 'pg_stat_statements';
name | default_version | installed_version | comment
--------------------+-----------------+-------------------+-----------------------------------------------------------
pg_stat_statements | 1.7 | 1.7 | track execution statistics of all SQL statements executed
2.4 Plug-in use
Directly query the plug-in view to see the TOP SQL situation:
postgres=# select * from pg_stat_statements limit 1;
-[ RECORD 1 ]-------+------------------------------------------------------------------------
userid | 10 //用户id
dbid | 13547 //数据库oid
queryid | 1194713979 //查询id
query | SELECT * FROM pg_available_extensions WHERE name = 'pg_stat_statements' //查询SQL
calls | 1 //调用次数
total_time | 53.363875 //SQL总共执行时间
min_time | 53.363875 //SQL最小执行时间
max_time | 53.363875 //SQL最大执行时间
mean_time | 53.363875 //SQL平均执行时间
stddev_time | 0 //SQL花费时间的表中偏差
rows | 1 //SQL返回或者影响的行数
shared_blks_hit | 1 //SQL在在shared_buffer中命中的块数
shared_blks_read | 0 //SQL从page cache或者磁盘中读取的块数
shared_blks_dirtied | 0 //SQL语句弄脏的shared_buffer的块数
shared_blks_written | 0 //SQL语句写入的块数
local_blks_hit | 0 //临时表中命中的块数
local_blks_read | 0 //临时表需要读的块数
local_blks_dirtied | 0 //临时表弄脏的块数
local_blks_written | 0 //临时表写入的块数
temp_blks_read | 0 //从临时文件读取的块数
temp_blks_written | 0 //从临时文件写入的数据块数
blk_read_time | 0 //从磁盘或者读取花费的时间
blk_write_time | 0 //从磁盘写入花费的时间
3. Slow SQL troubleshooting methods
3.1 Query the current session
SELECT pgsa.datname AS database_name
, pgsa.usename AS user_name
, pgsa.client_addr AS client_addr
, pgsa.application_name AS application_name
, pgsa.state AS state
, pgsa.backend_start AS backend_start
, pgsa.xact_start AS xact_start
, extract(epoch FROM now() - pgsa.xact_start) AS xact_time, pgsa.query_start AS query_start
, extract(epoch FROM now() - pgsa.query_start) AS query_time
, pgsa.query AS query_sql
FROM pg_stat_activity pgsa
WHERE pgsa.state != 'idle'
AND pgsa.state != 'idle in transaction'
AND pgsa.state != 'idle in transaction (aborted)'
ORDER BY query_time DESC
LIMIT 5;
3.2 View TOP SQL
Restart and reset the pg_stat_statements plugin to allow the plugin to re-collect current data:
create extension pg_stat_statements;
select pg_stat_reset();
select pg_stat_statements_reset();
View TOP SQL:
-- 总查询时间 TOP
select * from pg_stat_statements order by total_time desc limit 5;
-- 总 IO 消耗 TOP
select * from pg_stat_statements order by blk_read_time+blk_write_time desc limit 5;
-- 总调用次数 TOP
select * from pg_stat_statements order by calls desc limit 5;
When an abnormal session is found, it is usually killed first, and then the cause of the problem is analyzed. For related operations of kill session, you can refer to: "PostgreSQL Session Management"