[Day 23] SQL Advanced - Query Optimization - performance_schema Series V: Database Object Events and Attribute Statistics (SQL Xiao Xuzhu)

Teleportation back to the city – "32 Days SQL Foundation Building"

Zero. Foreword

Today is the 23rd day of learning SQL check-in , and every day I will provide an article for group members to read ( no need to pay for subscription ).

I hope that everyone will think for themselves first, if you really have no idea, then read the following problem-solving ideas, and implement it again by yourself. In the Xiaoxuzhu JAVA community , the corresponding [check-in stickers] check-in, today's task is completed, and develop a good habit of learning to check-in every day.

​ Brother Xuzhu will organize everyone to study the same article together, so if you have any questions, you can ask them in the group. Friends in the group can help you quickly. One person can go fast, and a group of people can go very fast. How lucky it is to have comrades-in-arms who study and communicate together.

​ My learning strategy is very simple, question sea strategy + Feynman learning method. If you can implement all these questions seriously, it means that SQL has successfully established its foundation. For the advanced learning later, you can continue to follow me and walk towards the road of architect together.

Today's learning content is: SQL Advanced - Query Optimization - performance_schema Series Five: Database Object Events and Attribute Statistics

The content of this article is more than 10,000 words, and there is a lot of content. Suggested reading method: remember the title and the introduction of the small modules. The detailed explanation of the content can help you understand. It’s okay if you don’t understand it.
This article is suitable for collection. When you need to consult the specified configuration, you can quickly find the detailed explanation of the configuration.

1. Exercise topics

topic link	difficulty
-	-

Two, SQL ideas

SQL Advanced - Query Optimization - performance_schema Series Five: Database Object Events and Attribute Statistics

After studying the performance_schema event statistics table yesterday, today I will learn more fine-grained classification statistics, such as table io overhead, lock overhead, and some attribute statistics of user connections. These need to look at the database object event statistics table and attribute statistics surface.

Database object event statistics table

Database table level object wait event statistics

Count waiting events by database object name (library name, table name, etc.).

Statistics are stored in the table objects_summary_global_by_type

 use performance_schema;

select * from objects_summary_global_by_type where SUM_TIMER_WAIT!=0;

Simple analysis:
according to the user_info_test table of the "studymysql" library, we grouped and
counted the number of wait event calls related to the table, the total, minimum, average, and maximum delay time information. Using this information, we can roughly understand the ranking of access efficiency of tables in InnoDB Statistics, to a certain extent, reflect the efficiency of calling the storage engine interface.

Table I/O wait and lock wait event statistics

Similar to the database table-level object wait event statistics table, the table I/O wait and lock wait event statistics are more refined, subdivided: subdivided the execution times of additions, deletions, changes, and queries for each table, the total waiting time, minimum, maximum , the average waiting time, and even down to the waiting time of adding, deleting, modifying and checking an index.

There are several tables like this:

show tables like '%table%summary%';

| table_io_waits_summary_by_index_usage | # Table I/O wait events counted by each index | table_io_waits_summary_by_table | # Table I/O wait events
counted by each table
| table_lock_waits_summary_by_table | # Table lock wait events counted by each table

Statistical Table Statistical Content View

select * from table_io_waits_summary_by_index_usage where SUM_TIMER_WAIT!=0;

select * from table_io_waits_summary_by_table where SUM_TIMER_WAIT!=0;

select * from table_lock_waits_summary_by_table where SUM_TIMER_WAIT!=0;

The records in this statistical table are too long, you can check the complete results by yourself according to the sql provided above.

As shown above:

• The table_io_waits_summary_by_table table contains the addition, deletion, modification, query and classification statistics of waiting events for the entire table

• table_io_waits_summary_by_index_usage distinguishes the addition, deletion, modification, and query of the indexes of each table and waits for event classification statistics

• The table_lock_waits_summary_by_table table statistics add, delete, modify and check the corresponding lock waiting time, not the IO waiting time

Focus on these three tables:

table_io_waits_summary_by_table表：

This table allows the use of the TRUNCATE TABLE statement. Only resets statistics columns to zero, not deletes rows. Executing truncate on this table also implicitly truncate table_io_waits_summary_by_index_usage table

table_io_waits_summary_by_index_usage表：

According to the grouping column + INDEX_NAME column of table_io_waits_summary_by_table, INDEX_NAME has the following types:

• If the index is used, the name of the index is displayed here. If it is PRIMARY, it means that the table I/O uses the primary key index

• If the value is NULL, it means that the table I/O does not use the index

• If it is an insert operation, the index cannot be used, and the statistical value at this time is calculated according to INDEX_NAME = NULL
  This table allows the use of the TRUNCATE TABLE statement. Only resets statistics columns to zero, not deletes rows.

When using DDL statements to change the index structure, it will cause all index statistics of the table to be reset.

table_lock_waits_summary_by_table表：

The grouping columns of this table are the same as table_io_waits_summary_by_table
This table contains information about internal and external locks:

• Internal locks correspond to locks in the SQL layer. It is achieved by calling the thr_lock() function.

• External locks correspond to locks in the storage engine layer. This is achieved by calling the handler::external_lock() function.
  This table allows the use of the TRUNCATE TABLE statement. Only resets statistics columns to zero, not deletes rows.

File I/O event statistics

The file I/O event statistics table only records IO events in waiting events (excluding table and socket subcategories), and there are several tables:

 show tables like '%file_summary%';

file_summary_by_event_name: File IO waiting events counted according to each event name file_summary_by_instance: File IO waiting events
counted according to each file instance (corresponding to each specific disk file, for example: table space file sbtest1.ibd of table sbtest1)

Statistical Table Statistical Content View

 select * from file_summary_by_event_name where SUM_TIMER_WAIT !=0 and EVENT_NAME like '%innodb%' limit 1;
select * from file_summary_by_instance where SUM_TIMER_WAIT!=0 and EVENT_NAME like '%innodb%' limit 1;

Description of common statistical fields:

• COUNT_STAR, SUM_TIMER_WAIT, MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT: These columns count the number and operation time of all I/O operations;
• COUNT_READ, SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ, SUM_NUMBER_OF_BYTES_READ: These columns count all file read operations, including FGETS, FGETC, FREAD and READ system calls, and also include the number of data bytes for these I/O operations;
• COUNT_WRITE, SUM_TIMER_WRITE, MIN_TIMER_WRITE, AVG_TIMER_WRITE, MAX_TIMER_WRITE, SUM_NUMBER_OF_BYTES_WRITE: These columns count all file write operations, including FPUTS, FPUTC, FPRINTF, VFPRINTF, FWRITE and PWRITE system calls, and also include the data bytes of these I/O operations;
• COUNT_MISC, SUM_TIMER_MISC, MIN_TIMER_MISC, AVG_TIMER_MISC, MAX_TIMER_MISC: These columns count all other file I/O operations, including CREATE, DELETE, OPEN, CLOSE, STREAM_OPEN, STREAM_CLOSE, SEEK, TELL, FLUSH, STAT, FSTAT, CHSIZE, RENAME, and SYNC system call. Note: There is no byte count information for these file I/O operations.
• Note: The TRUNCATE TABLE statement is allowed. But only reset the stat column to zero, not delete the row.

Socket event statistics

Socket events count the number of socket read and write calls and send and receive byte count information. There are several tables:

show tables like '%socket%summary%';

socket_summary_by_instance: For all socket I/O operations of each socket instance, the number of operations, time, and sent and received byte information related to these socket operations are generated by wait/io/socket/* instruments. But when the connection is interrupted, the information row corresponding to the socket connection in the table will be deleted (the socket here refers to the socket instance created by the current active connection)
socket_summary_by_event_name: For each socket I/O instruments, these socket operations are related The number of operations, time, and sent and received byte information are generated by wait/io/socket/* instruments (the socket here refers to the socket instance created by the current active connection)

Statistical Table Statistical Content View

select * from socket_summary_by_event_name;

select * from socket_summary_by_instance where COUNT_STAR!=0;

Description of common statistical fields:

• COUNT_STAR, SUM_TIMER_WAIT, MIN_TIMER_WAIT, AVG_TIMER_WAIT, MAX_TIMER_WAIT: These columns count the number and time information of all socket read and write operations

• COUNT_READ, SUM_TIMER_READ, MIN_TIMER_READ, AVG_TIMER_READ, MAX_TIMER_READ, SUM_NUMBER_OF_BYTES_READ: These columns count all receiving operations (socket RECV, RECVFROM, RECVMS type operations, that is, the operation of socket reading data with the server as a reference) related times, time, received word Section number and other information

• COUNT_WRITE, SUM_TIMER_WRITE, MIN_TIMER_WRITE, AVG_TIMER_WRITE, MAX_TIMER_WRITE, SUM_NUMBER_OF_BYTES_WRITE: These columns count the number, time, and reception of all sending operations (socket SEND, SENDTO, SENDMSG type operations, that is, the operation of writing data to the socket with the server as a reference) information such as the number of bytes

• COUNT_MISC, SUM_TIMER_MISC, MIN_TIMER_MISC, AVG_TIMER_MISC, MAX_TIMER_MISC: These columns count all other socket operations, such as socket CONNECT, LISTEN, ACCEPT, CLOSE, SHUTDOWN type operations. NOTE: These operations do not have byte counts

• NOTE: The TRUNCATE TABLE statement is allowed (except events_statements_summary_by_digest), which only resets the statistics column to zero, not deletes the row

prepare statement instance statistics table

What is prepare statement? The prepare statement is actually a precompiled statement. First compile the SQL statement, and you can set a parameter placeholder (for example: ? symbol), and then pass in the specific parameter value through the user variable when calling. The prepare statement has three steps, precompile the prepare statement, execute the prepare statement, release and destroy the prepare statement.
performance_schema provides monitoring records for prepare statements.

Statistical Table Statistical Content View

select * from prepared_statements_instances;

No data, can't see the field. Brother Xu Zhu, let me teach you how to use prepared statements by the way.

combat

 PREPARE stmt1 FROM 'SELECT SQRT(POW(?,2) + POW(?,2)) AS hypotenuse';
SET @a = 3;
SET @b = 4;
EXECUTE stmt1 USING @a, @b;

In the first step, use the PREPARE statement to prepare the statement for execution. This statement is used to calculate the hypotenuse of a triangle given the lengths of the two sides.
In the second step, two variables @a and @b are declared;
the third step: third, use the EXECUTE statement to execute the prepared statement of the variables @a and @b.
Then we can see the statistics.

Fourth, we use DEALLOCATE PREPARE to release resources.

DEALLOCATE PREPARE stmt1;

As shown above:

• prepare statement precompilation: Create a prepare statement. If the statement detection is successful, a new row will be added to the prepared_statements_instances table.
• Prepare statement execution: The execution of the EXECUTE statement is detected, and the corresponding row information in the prepare_statements_instances table will be updated.
• The prepare statement releases resource allocation: The detected prepare statement instance executes the COM_STMT_CLOSE or SQLCOM_DEALLOCATE_PREPARE command, and the corresponding row information in the prepare_statements_instances table will be deleted at the same time. To avoid resource leaks, be sure to perform this step to release resources when the prepare statement is not needed.

field description

• OBJECT_INSTANCE_BEGIN: The memory address of the instruments instance of the prepare statement event.

• STATEMENT_ID: The internal ID of the statement assigned by the server. Both text and binary protocols use this statement ID.

• STATEMENT_NAME: For statement events of the binary protocol, the value of this column is NULL. For statement events for text protocols, this column value is the user-assigned external statement name. For example: PREPARE stmt FROM'SELECT 1';, the statement name is stmt.

• SQL_TEXT: The statement text of prepare, with "?" indicates that it is a placeholder mark, and the subsequent execute statement can pass parameters to this mark.

• OWNER_THREAD_ID, OWNER_EVENT_ID: These columns represent the thread ID and event ID that created the prepare statement.

• OWNER_OBJECT_TYPE, OWNER_OBJECT_SCHEMA, OWNER_OBJECT_NAME: For prepare statements created directly by client sessions using SQL statements, these column values ​​are NULL. For prepare statements created by stored procedures, these column values ​​display information about the associated stored procedure. If the user forgets to release the prepare statement in the stored program, these columns can be used to find the stored program corresponding to the unreleased prepare, and use the statement query:

SELECT OWNER_OBJECT_TYPE,OWNER_OBJECT_SCHEMA,OWNER_OBJECT_NAME,STATEMENT_NAME,SQL_TEXT FROM performance_schema.prepared_statemments_instances WHERE OWNER_OBJECT_TYPE IS NOT NULL;

• TIMER_PREPARE: The time spent executing the prepare statement itself.
• COUNT_REPREPARE: The number of times the prepare statement corresponding to the row information is recompiled internally. After the prepare statement is recompiled, the previous relevant statistical information is no longer available, because these statistical information are aggregated into the table as part of the statement execution. rather than maintained separately.
• COUNT_EXECUTE, SUM_TIMER_EXECUTE, MIN_TIMER_EXECUTE, AVG_TIMER_EXECUTE, MAX_TIMER_EXECUTE: related statistical data when executing the prepare statement.
• SUM_xxx: The remaining columns beginning with SUM_xxx are the same as the information in the statement statistics table
• Note: The TRUNCATE TABLE statement is allowed, but TRUNCATE TABLE only resets the statistics columns of the prepared_statements_instances table, but does not delete the records in the table, which will be automatically deleted when the prepare object is destroyed and released.

lock object record table

performance_schema records related lock information through the following table:

• metadata_locks: metadata lock holding and request records;
• table_handles: Table lock holding and request records.

metadata_locks table

Performance Schema records metadata lock information through the metadata_locks table:

• Granted locks (shows which sessions own the current metadata lock);
• Locks requested but not granted (shows which sessions are waiting for which metadata locks);
• A lock that has been detected and killed by a deadlock detector, or a lock request timeout waiting for a lock request session is dropped.

The metadata_locks table is read-only and cannot be updated. The default number of reserved rows will be automatically adjusted. If you want to configure the size of the table, you can set the value of the system variable performance_schema_max_metadata_locks before starting the server.

Statistical Table Statistical Content View

select * from metadata_locks;

Field description:

• OBJECT_TYPE: The lock type used in the metadata lock subsystem (similar to the OBJECT_TYPE column value in the setup_objects table): Valid values ​​are: GLOBAL, SCHEMA, TABLE, FUNCTION, PROCEDURE, TRIGGER (not currently used), EVENT, COMMIT, USER LEVEL LOCK, TABLESPACE, LOCKING SERVICE, USER LEVEL LOCK values ​​indicate that the lock is acquired using the GET_LOCK() function. The LOCKING SERVICE value indicates the lock acquired using the lock service;
• OBJECT_SCHEMA: Which library-level object does the lock come from;
• OBJECT_NAME: the name of the instruments object, a table-level object;
• OBJECT_INSTANCE_BEGIN: the memory address of the instruments object;
• LOCK_TYPE: The lock type in the metadata lock subsystem. Valid values ​​are: INTENTION_EXCLUSIVE, SHARED, SHARED_HIGH_PRIO, SHARED_READ, SHARED_WRITE, SHARED_UPGRADABLE, SHARED_NO_WRITE, SHARED_NO_READ_WRITE, EXCLUSIVE;
• LOCK_DURATION: The lock duration from the metadata lock subsystem. Valid values ​​are: STATEMENT, TRANSACTION, EXPLICIT, where the STATEMENT and TRANSACTION values ​​represent locks that will be released at the end of the statement or transaction, respectively. The EXPLICIT value indicates a lock that can be retained at the end of a statement or transaction and needs to be explicitly released, for example: a global lock acquired using FLUSH TABLES WITH READ LOCK;
• LOCK_STATUS: The lock status of the metadata lock subsystem. Valid values ​​are: PENDING, GRANTED, VICTIM, TIMEOUT, KILLED, PRE_ACQUIRE_NOTIFY, POST_RELEASE_NOTIFY. performance_schema changes the lock state to these values ​​according to different stages;
• SOURCE: the name of the source file, which contains the detection code line number that generated the event information;
• OWNER_THREAD_ID: ID of the thread requesting the metadata lock;
• OWNER_EVENT_ID: The event ID that requested the metadata lock.

How performance_schema manages the content recorded in the metadata_locks table (using the LOCK_STATUS column to indicate the status of each lock):

• When requesting immediate acquisition of a metadata lock, a lock information row with status GRANTED will be inserted;
• When the requested metadata lock cannot be obtained immediately, a lock information row with status PENDING will be inserted;
• When a lock that cannot be obtained immediately after the previous request is granted, the status of the lock information row is updated to GRANTED;
• When the metadata lock is released, the corresponding lock information row is deleted;
• When a pending state lock is detected by the deadlock detector and selected to break the deadlock, the lock will be canceled and an error message (ER_LOCK_DEADLOCK) will be returned to the session requesting the lock, and the lock state will be updated from PENDING to VICTIM;
• When the pending lock request times out, an error message (ER_LOCK_WAIT_TIMEOUT) will be returned to the session requesting the lock, and the lock status will be updated from PENDING to TIMEOUT;
• When a granted lock or pending lock request is killed, its lock status is updated from GRANTED or PENDING to KILLED;
• The status values ​​of VICTIM, TIMEOUT and KILLED stay for a short time. When a lock is in this state, it means that the lock row information is about to be deleted (manually executing SQL may not be visible due to time reasons, and you can use the program to grab it);
• The PRE_ACQUIRE_NOTIFY and POST_RELEASE_NOTIFY state value stay events are very short. When a lock is in this state, it means that the metadata lock subsystem is notifying the relevant storage engine that the lock is being allocated or released. These status values ​​were added in version 5.7.11.
• Note: The metadata_locks table does not allow the TRUNCATE TABLE statement.

table_handles table

performance_schema records table lock information through the table_handles table to track and record the table locks currently held by each open table. table_handles outputs the content collected by table lock instruments. This information shows which tables are open in the server, what the locking method is, and which session is holding it.

The table_handles table is read-only and cannot be updated. By default, the table data row size is automatically adjusted. If you want to specify it explicitly, you can set the value of the system variable performance_schema_max_table_handles before the server starts.

Statistical Table Statistical Content View

select * from table_handles;

Field description:

• OBJECT_TYPE: Displays the type of handles lock, indicating which table handles the table is opened by;
• OBJECT_SCHEMA: Which library-level object does the lock come from;
• OBJECT_NAME: the name of the instruments object, a table-level object;
• OBJECT_INSTANCE_BEGIN: the memory address of the instruments object;
• OWNER_THREAD_ID: ID of the thread holding the handles lock;
• OWNER_EVENT_ID: The event ID that triggers the table handles to be opened, that is, the event ID that holds the handles lock
• INTERNAL_LOCK: The table lock used at the SQL level. Valid values ​​are: READ, READ WITH SHARED LOCKS, READ HIGH PRIORITY, READ NO INSERT, WRITE ALLOW WRITE, WRITE CONCURRENT INSERT, WRITE LOW PRIORITY, WRITE.
• EXTERNAL_LOCK: The table lock used at the storage engine level. Valid values ​​are: READ EXTERNAL, WRITE EXTERNAL.
• Note: The TRUNCATE TABLE statement is not allowed for the table_handles table.

attribute statistics table

Connection Information Statistics Table

When a client connects to a MySQL server, its username and hostname are specified. performance_schema classifies the statistical information of these connections according to account, host, and user name and saves them in the connection information table of each category, as follows:

• accounts: count the connections of each client in the form of user@host;
• hosts: count each client connection according to the host name;
• users: Count each client connection by user name.

accounts table

The accounts table contains a record for each account connected to the MySQL server. For each account, each user+host uniquely identifies a row, and each row independently calculates the current number of connections and the total number of connections of the account. When the server starts, the size of the table is automatically adjusted. To explicitly set the table size, you can set the value of the system variable performance_schema_accounts_size before the server starts. When this system variable is set to 0, it means that the statistical information function of the accounts table is disabled.

select * from accounts;

Field description:

• USER: The client user name of a connection. This field is NULL if the connection was created by an internal thread, or by an unauthenticated user;
• HOST: The client host name of a connection. This field is NULL if the connection was created by an internal thread, or by an unauthenticated user;
• CURRENT_CONNECTIONS: the current number of connections of an account;
• TOTAL_CONNECTIONS: The total number of connections of an account (a new connection adds up to one, and it will not decrease like the current number of connections if the connection is disconnected).

users table

The users table contains connection information for each user connected to the MySQL server, one row for each user. The table will count the current number of connections and the total number of connections based on the user name as the unique identifier. When the server starts, the size of the table will be automatically adjusted. To explicitly set the table size, set the value of the system variable performance_schema_users_size before the server starts. When this variable is set to 0, users statistics are disabled.

select * from users;

Field description:

• USER: The user name of a connection, if it is a connection created by an internal thread, or a connection created by an unauthenticated user, this field is NULL;
• CURRENT_CONNECTIONS: the current number of connections of a user;
• TOTAL_CONNECTIONS: The total number of connections for a user.

hosts table

The hosts table contains the host information of the client connecting to the MySQL server. A host name corresponds to a row of records. This table counts the current number of connections and the total number of connections for the host as a unique identifier. When the server starts, the size of the table is automatically adjusted. To explicitly set the table size, set the value of the system variable performance_schema_hosts_size before the server starts. If this variable is set to 0, the hosts table statistics are disabled.

select * from hosts;

Field description:

• HOST: the host name of a connection, if it is a connection created by an internal thread, or a connection created by an unauthenticated user, this field is NULL;
• CURRENT_CONNECTIONS: the current number of connections to a host;
• TOTAL_CONNECTIONS: The total number of connections for a host.

Connection attribute statistics table

The application can use some key/value pairs to generate some connection properties, which are passed to the server when creating a connection to the mysql server. For the C API, use the mysql_options() and mysql_options4() functions to define attribute sets. Other MySQL connectors can use some custom connection property methods.
Connection properties are documented in the following two tables:

• session_account_connect_attrs: record the connection attributes of the current session and its associated other sessions;
• session_connect_attrs: Connection attributes for all sessions.

session_account_connect_attrs表

Applications can use the mysql_options() and mysql_options4() C API functions to provide key-value pair connection attributes to be passed to the server at connection time.
The session_account_connect_attrs table only contains connection attributes for the current connection and its associated other connections. To see connection attributes for all sessions, look in the session_connect_attrs table.

select * from session_account_connect_attrs;

Field description:

• PROCESSLIST_ID: The connection identifier of the session, which is the same as the ID field in the show processlist result;
• ATTR_NAME: connection attribute name;
• ATTR_VALUE: connection attribute value;
• ORDINAL_POSITION: The order in which connection properties are added to the connection property set.
• The session_account_connect_attrs table does not allow the TRUNCATE TABLE statement.

session_connect_attrs table

The table fields have the same meaning as the session_account_connect_attrs table, but this table is the connection attribute table for saving all connections.

 select * from session_connect_attrs;

The meanings of the table fields are the same as those of the session_account_connect_attrs table fields.

Summarize

This article explains the database object event and attribute statistics in performance_schema, and introduces in detail the database object event statistics table: database table level object wait event statistics table, table I/O wait and lock wait event statistics table, file I/O event statistics table, socket event statistics table, prepare statement instance statistics table, and lock object record table.
It also introduces the connection information statistics and connection attribute statistics, two kinds of attribute statistics tables.
The field descriptions and some precautions of these statistical tables are introduced in detail.

The content of this article is more than 10,000 words, and there is a lot of content. Suggested reading method: remember the title and the introduction of the small modules. The detailed explanation of the content can help you understand. It’s okay if you don’t understand it.
This article is suitable for collection. When you need to consult the specified configuration, you can quickly find the detailed explanation of the configuration.

4. Reference

Database object event and attribute statistics | performance_schema comprehensive introduction

I'm Brother Xuzhu, see you tomorrow~