Teleportation back to the city – "32 Days SQL Foundation Building"
Article Directory
- Zero. Foreword
- 1. Background
- 2. The performance_schema configuration configuration table enables the collection and recording of waiting events
- 3. sysbench benchmark tool
- 4. Sysbench pressurizes the mysql database
- V. Summary
- 6. Reference
Zero. Foreword
Today is the 24th day of learning SQL check-in . Every day I will provide an article for group members to read ( no need to pay for subscription ).
I hope that everyone will think for themselves first, if you really have no idea, then read the following problem-solving ideas, and implement it again by yourself. In the Xiaoxuzhu JAVA community , the corresponding [check-in stickers] check-in, today's task is completed, and develop a good habit of learning to check-in every day.
Brother Xuzhu will organize everyone to study the same article together, so if you have any questions, you can ask them in the group. Friends in the group can help you quickly. One person can go fast, and a group of people can go very fast. How lucky it is to have comrades-in-arms who study and communicate together.
My learning strategy is very simple, question sea strategy + Feynman learning method. If you can implement all these questions seriously, it means that SQL has successfully established its foundation. For the advanced learning later, you can continue to follow me and walk towards the road of architect together.
Today's learning content is: SQL Advanced - Query Optimization - Performance_schema Series Practice 1: Using Waiting Events to Troubleshoot MySQL Performance Problems
1. Background
Before the production goes online, benchmark the database for adding, deleting, modifying and checking, collect benchmark data, and prepare for subsequent expansion and architecture upgrade.
MySQL database benchmarks usually choose sysbench, tpcc-mysql, workbench.
The following uses the sysbench benchmark tool to test the MySQL database as an example to introduce how to use the wait events of performance_schema to troubleshoot the database performance bottleneck.
2. The performance_schema configuration configuration table enables the collection and recording of waiting events
Use the performance_schema configuration table to enable collection and logging of wait events.
use performance_schema;
Modify the enabled and timed fields of the setup_instruments table to yes, indicating that the corresponding instruments are enabled
update setup_instruments set enabled='yes',timed='yes' where name like 'wait/%';
View the modification results, if the enabled and timed fields are YES, it means that the current instruments have been enabled (but at this time the collector will not collect event data immediately, it is necessary to save the table of these waiting events – consumers, and the collection will start after enabling)
select * from setup_instruments where name like 'wait/%';
Enable consumers that wait for events
update setup_consumers set enabled='yes' where name like '%wait%';
search result:
select * from setup_consumers where name like '%wait%';
3. sysbench benchmark tool
The mysql benchmark test can be understood as a stress test for the runtime of the mysql database. There are three key indicators:
TPS/QPS: Measures throughput.
Response time: including average response time, minimum response time, maximum response time and time percentage Zhu.
Concurrency: The number of query requests processed at the same time.
sysbench supports multi-threaded work and can be installed and deployed across platforms.
3.1 Install and use sysbench
3.1.1 yum installation
curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash
sudo yum -y install sysbench
As shown in the picture, the installation is successful.
3.1.2 View version information
sysbench --version
3.1.3 Sysbench Instructions
sysbench --help
Values in square brackets below represent default values
| option type | parameter name | Parameter meaning |
|---|---|---|
| general option | threads | Specifies the number of threads [1] |
| general option | events | Limit the maximum number of requests, 0 means no limit[0] |
| general option | time | Limit the maximum execution time, 0 means no limit [10], unit second |
| general option | forced-shutdown | How long do you need to wait after reaching the maximum execution time? Close sysbench off means disable this function [off] |
| general option | thread-stack-size | The size of the stack space used by each thread [64K] |
| general option | rate | Average transaction processing rate, 0 means unlimited [0] |
| general option | report-interval | Report results every few seconds, 0 disables interval reporting[0] |
| general option | config-file | Read command-line options from a file |
| mysql-specific options | mysql-host | mysql hostname, [localhost] |
| mysql-specific options | mysql-port | mysql-port, [3306] |
| mysql-specific options | mysql-socket | Specify the socket file to connect |
| mysql-specific options | mysql-user | Username for logging in to mysql, default value: [sbtest] |
| mysql-specific options | mysql-password | Password to log in to mysql |
| mysql-specific options | mysql-db | Specify the database name, default: [sbtest] |
| mysql-specific options | mysql-ssl | connect using ssl |
| mysql-specific options | mysql-ssl-cipher | password when using ssl connection |
| mysql-specific options | mysql-compression | use compression algorithm |
| mysql-specific options | mysql-debug | track all client usage, [off] |
| mysql-specific options | mysql-ignore-errors | Ignore the specified error code or use all to ignore all errors, [1213,1020,1205] |
-
testname : Specifies the name of the test to be run.
-
command: Represents the command to be executed by sysbench, including prepare, run and cleanup.
- prepare : Prepare data in advance for testing.
- run : Execute formal tests.
- cleanup: clean up the database after the test is completed.
3.2 sysbench test server cpu performance
sysbench cpu --cpu-max-prime=20000 --threads=2 run
The test for cpu is to perform the addition test of prime numbers, among which:
–cpu-max-prime: the upper limit of the number of prime numbers generated.
–threads: Start the number of threads for prime calculation.
illustrate:
Running the test with following options:
#指定线程个数
Number of threads: 2
Initializing random number generator from current time
#每个线程产生的素数上限为2万个。
Prime numbers limit: 20000
Initializing worker threads...
Threads started!
CPU speed:
#所有线程每秒完成了839.89次event
events per second: 839.89
General statistics:
total time: 10.0022s
total number of events: 8402 #在10.0022s秒内共完成了8402次event
Latency (ms):
min: 2.29
avg: 2.38
max: 10.31
95th percentile: 2.61 #95%的events都在2.61毫秒内完成。
sum: 19997.47
Threads fairness:
# 平均每完成4201.0000次event,标准差是2
events (avg/stddev): 4201.0000/2.00
# 每个线程平均耗时9.9987秒。
execution time (avg/stddev): 9.9987/0.00
3.3 Sysbench test hard disk IOPS
3.3.1 Prepare test data
sysbench fileio --file-total-size=1G --file-test-mode=rndrw --time=30 --max-requests=0 prepare
3.3.2 Start the test
sysbench fileio --file-total-size=1G --file-test-mode=rndrw --time=30 --max-requests=0 run
Execution effect:
The formula for calculating IOPS is as follows:
IOPS=(Throughput read +Throughput written)*1024/16KB
Throughput read: Input per second
Throughput written: Output per second.
From the test data in the figure, the current IOPS of the hard disk can be calculated as:
IOPS=（ 12.31+8.21）*1024/16=1313.28
3.3.3 Clear test data
sysbench fileio --file-total-size=1G --file-test-mode=rndrw --time=30 --max-requests=0 cleanup
3.4 Combat: Using sysbench to test the mysql database
sysbench provides lua scripts for database performance testing. These scripts are placed in: /usr/share/sysbench/directory.
In order to view the structure in tree form, first install the tree command.
sudo yum install -y tree
List lua scripts.
tree /usr/share/sysbench/ -P *.lua
The following actual combat demonstrates the use of lua scripts to test the mysql database.
3.4.1 Prepare test data
Create a test database sysbenchdemo
create database sysbenchdemo;
Prepare test data:
sysbench /usr/share/sysbench/oltp_insert.lua \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-socket=/tmp/mysql.sock \
--mysql-user=root \
--mysql-password=xiaoxuzhu \
--mysql-db=sysbenchdemo \
--db-driver=mysql \
--tables=5 \
--table-size=100000 \
--time=180 prepare
illustrate:
Use the /usr/share/sysbench/oltp_insert.lua script to create 5 tables and insert 100,000 pieces of data into each table.
3.4.2 Start the test
sysbench /usr/share/sysbench/oltp_insert.lua \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-socket=/tmp/mysql.sock \
--mysql-user=root \
--mysql-password=xiaoxuzhu \
--mysql-db=sysbenchdemo \
--db-driver=mysql \
--tables=5 \
--table-size=100000 \
--time=180 run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)
Running the test with following options:
Number of threads: 1
Initializing random number generator from current time
Initializing worker threads...
Threads started!
SQL statistics:
queries performed:
read: 0
# 总的写次数
write: 29970
other: 0
total: 29970
# 总的事务数和每秒事务数
transactions: 29970 (166.50 per sec.)
queries: 29970 (166.50 per sec.)
ignored errors: 0 (0.00 per sec.)
reconnects: 0 (0.00 per sec.)
General statistics:
# 总的执行时间和事件数。
total time: 180.0018s
total number of events: 29970
# 延时统计信息
Latency (ms):
min: 3.16
avg: 6.00
max: 237.40
95th percentile: 11.24
sum: 179912.58
Threads fairness:
events (avg/stddev): 29970.0000/0.00
execution time (avg/stddev): 179.9126/0.00
3.4.3 Clean test data
sysbench /usr/share/sysbench/oltp_insert.lua \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-socket=/tmp/mysql.sock \
--mysql-user=root \
--mysql-password=xiaoxuzhu \
--mysql-db=sysbenchdemo \
--db-driver=mysql \
--tables=5 \
--table-size=100000 \
--time=180 cleanup
4. Sysbench pressurizes the mysql database
Gradually increase the number of concurrent threads until tps and qps no longer increase with the number of threads
4.1 Prepare test data:
sysbench /usr/share/sysbench/oltp_insert.lua \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-socket=/tmp/mysql.sock \
--mysql-user=root \
--mysql-password=xiaoxuzhu \
--mysql-db=sysbenchdemo \
--db-driver=mysql \
--tables=8 \
--table-size=100000 \
--time=180 prepare
4.2 Start the test
sysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua \
--mysql-host=localhost \
--mysql-port=3306 \
--mysql-socket=/tmp/mysql.sock \
--mysql-user=root \
--mysql-password=xiaoxuzhu \
--mysql-db=sysbenchdemo \
--db-driver=mysql \
--oltp-table-size=5000000 \
--oltp-tables-count=8 \
--num-threads=16 \
--max-time=1800 \
--max-requests=0 \
--report-interval=1 run
From the output of sysbench, we can see that under the pressure of 16 concurrent threads oltp, tps can run to 600-900, qps: about 15000, and the delay is 1000ms+
why the delay is so high, use top to see, we can see The CPU usage of mysql is high.
In order to facilitate the query of waiting event statistics, we can first create a view for real-time statistics of current waiting events (non-historical data)
use performance_schema;
create view sys.test_waits as select sum(TIMER_WAIT) as TIMER_WAIT,sum(NUMBER_OF_BYTES) as NUMBER_OF_BYTES, EVENT_NAME,OPERATION from events_waits_current where EVENT_NAME!='idle' group by EVENT_NAME,OPERATION;
select sys.format_time(TIMER_WAIT),sys.format_bytes(NUMBER_OF_BYTES),EVENT_NAME,OPERATION from sys.test_waits where sys.format_time(TIMER_WAIT) not regexp 'ns|us' order by TIMER_WAIT desc;
You can also directly query the events_waits_current table
select THREAD_ID,EVENT_NAME,sys.format_time(TIMER_WAIT),INDEX_NAME,NESTING_EVENT_TYPE,OPERATION,NUMBER_OF_BYTES
from events_waits_current
where EVENT_NAME!='idle' order by TIMER_WAIT desc;
From the query results of the above waiting events, the cpu is not enough, and the wait/synch/cond/mysqlx/scheduler_dynamic_worker_pending scheduler dynamic worker program is suspended.
V. Summary
This article shares how to install and use sysbench, introduces in detail how to use sysbench to pressure test the mysql database, and use waiting events to troubleshoot MySQL performance problems.
Hahaha, have you learned~
6. Reference
A collection of application examples | performance_schema comprehensive introduction (on)
[Mysql database advanced practice - Chapter 10 mysql performance optimization and operation and maintenance management] Author: Teacher Zhao Yuqiang
I'm Brother Xuzhu, see you tomorrow~