Five dimensions of Mysql performance optimization

Five dimensions of Mysql performance optimization

1. Database connection configuration optimization

1. What the server needs to do is to accept as many client connections as possible. Maybe you have encountered error 1040: Too many connections? It's just that the number of connections on the server side is not enough.
2. What the client can do is to minimize the number of times it establishes connections with the server. Use the already established connections if they can be used. Do not create a new connection every time you execute a SQL statement. The resources of both the server and the client will be overwhelmed. .

1. Optimized configuration of the maximum number of server connections

What the server needs to do is to accept as many client connections as possible. Maybe you have encountered error 1040: Too many connections? It's just that the number of connections on the server side is not enough.
What the client can do is to minimize the number of times it establishes a connection with the server. Use the existing connections if they can be used. Do not create a new connection every time a SQL statement is executed. The resources of both the server and the client will be overwhelmed.

The solution is to use a connection pool to reuse connections.
The problem of insufficient connections can be solved from two aspects:
(1) Method 1: Increase the number of available connections and modify the environment variable max_connections. By default, the maximum number of connections on the server is 151
mysql> show variables like 'max_connections';
± ----------------±------+
| Variable_name | Value |
±----------------±--- ---+
| max_connections | 151 |
±----------------±------+
1 row in set (0.01 sec)

set GLOBAL max_connections=1000; #Modify the maximum number of connections to 1000

(2) Method 2
/etc/my.cnf #Add the configuration
max_connections=1500 to the database configuration file #Modify the maximum number of connections to 1500
and then restart the database.

2. Release inactive connections

Release inactive connections in a timely manner. The system's default client timeout is 28800 seconds (8 hours). We can adjust this value smaller
mysql> show variables like 'wait_timeout';
±---------- ----±------+
| Variable_name | Value |
±--------------±------+
| wait_timeout | 28800 |
±---- ----------±------+
1 row in set (0.01 sec)

Modify the configuration file
/etc/my.cnf
interactive_timeout=18800
wait_timeout=18800

Restart database

2. Architecture optimization

1. Use cache

(1) If the data is not particularly effective (it does not change every moment, such as daily reports), we can put this kind of data into the cache system, and during the cache validity period of the data, directly retrieve it from the cache system. Obtain data from the database, which can reduce the pressure on the database and improve query efficiency.

2. Separation of reading and writing

(1) You can use multiple database servers at the same time. Set one of them as the team leader, called the master node, and the other nodes as team members, called slaves. Users write data only to the master node, and read requests are distributed to various slave nodes. This solution is called read-write separation.

(2) Using a cluster will inevitably face a problem, master-slave replication to maintain data consistency between multiple nodes.
Binlog is the core component that implements the MySQL master-slave replication function.
①. The slave-side IO thread sends a request to the master-side binlog (binary log) thread.
②. The master-side binlog dump thread obtains the binary log information (file name and location information) and sends it to the slave-side IO thread.
③. The content obtained by the slave-side IO thread. Write to the slave side relay log (relay log) in turn, and record the bin-log file name and location of the master side to http://master.info ④. When the
SQL thread on the slave side detects that the content in the relay log is updated, it It will parse the updated content in the relay log and perform these operations to achieve consistency with the master data.

3. Sub-library and sub-table

Summary of database and table sharding: Summary: Horizontal sharding is mainly to solve storage bottlenecks; vertical sharding is mainly to reduce concurrency pressure.

(1) Vertical sharding:
The meaning of nodes in sharding databases and sharding tables is relatively broad. If the database is used as a node, it is a sharding database; if a single table is used as a node, it is a sharding table.
Sub-library and table sub-division are divided into vertical sub-database, vertical sub-table, horizontal sub-database and horizontal sub-table.

On the basis of a single database, make several vertical cuts and split it into different databases according to business logic. This is vertical sub-database.

(2) Vertical table splitting
Vertical table splitting is to cut one (or several cuts) vertically on a single table to split the multiple words of a table into several small tables. This operation needs to be carried out according to the specific business. Judgment, usually the frequently used fields (hot fields) are divided into a table, and the fields that are not frequently used or not used immediately (cold fields) are divided into a table to improve the query speed.

(3) Horizontal table splitting:
Save the data of a single table to multiple data tables according to certain rules (called sharding rules in jargon), and apply one knife (or several knifes) to the data table horizontally, which is horizontal table partitioning.

(4) Horizontal sub-database
Horizontal sub-database is to cut a single database horizontally, which is often accompanied by horizontal table splitting.

4. Message queue

Normally, user requests will directly access the database. If the number of online users at the same time is very large, it is very likely to overwhelm the database (refer to the status of Weibo when celebrities cheat or announce their relationships).
In this case, the pressure on the database can be reduced by using the message queue. No matter how many user requests there are at the same time, they are first stored in the message queue, and then the system consumes the requests from the message queue in an orderly manner.

3. SQL analysis and optimization

1. Slow query

Slow queries are queries that execute very slowly. Only by knowing what slow queries are there in MySQL can we perform targeted optimization.
Because turning on the slow query log has a performance cost, MySQL turns off the slow query log function by default. Use the following command to view the current slow query status.
show variables like 'slow_query%';
mysql> show variables like 'slow_query%';
±--------------------±---------- -------------------------------+
| Variable_name | Value |
±---------------- ----±----------------------------------------+
| slow_query_log | OFF |
| slow_query_log_file | /var/lib/mysql/9e74f9251f6c-slow.log |
±--------------------±-------------- -----------------------+
2 rows in set (0.00 sec)
slow_query_log indicates whether the slow query log is currently enabled, and slow_query_log_file indicates the storage location of the slow query log.
In addition to the above two variables, we also need to determine what the "slow" indicator is, that is, how long it takes to execute a query to be considered a slow query. The default is 10S. If it is changed to 0, all SQL show variables like '%long_query% will be recorded
. ';
mysql> show variables like ‘%long_query%’;
±----------------±----------+
| Variable_name | Value |
±----------------±----------+
| long_query_time | 10.000000 |
±----------------±----------+
1 row in set (0.00 sec)

2. Open the slow query log

(1) Method 1: Modify the configuration file my.cnf.
This modification will still be effective after the system restarts.

#Whether to enable slow query log
slow_query_log=ON
long_query_time=2
slow_query_log_file=/var/lib/mysql/slow.log

(2) Method 2: Dynamically modify parameters (invalid after restart)
mysql> set @@global.slow_query_log=1;
Query OK, 0 rows affected (0.06 sec)
mysql> set @@global.long_query_time=2;
Query OK, 0 rows affected (0.00 sec)

3. Slow query log analysis

MySQL not only saves slow log files for us, but also provides us with the slow log query tool mysqldumpslow. In order to demonstrate this tool, we first construct a slow query: mysql> SELECT sleep(5); View the number of slow query log
records
:
SHOW GLOBAL STATUS LIKE '%Slow_queries%';
Then we query the slow query that takes the most time:

mysqldumpslow -st -t 1 -g 'select' /var/lib/mysql/9e74f9251f6c-slow.log
Reading mysql slow query log from /var/lib/mysql/9e74f9251f6c-slow.log
Count: 1 Time=10.00s (10s ) Lock=0.00s (0s) Rows=1.0 (1), root[root]@localhost
SELECT sleep(N)
Among them
, Count: indicates the number of times this SQL is executed.
Time: indicates the execution time. The cumulative time in brackets is
Locks: Indicates the locking time, and the one in parentheses is the cumulative time.
Rows: Indicates the number of records returned, and the one in parentheses is the cumulative number.

cat /var/lib/mysql/data/slow.log
SHOW profiles View the time consumption of SQL
Query SQL The time consumption of the entire life cycle
can be obtained through Query_ID. The complete life of the four-layer structure from connection - service - engine - storage Cycle time
SHOW profile CPU, BLOCK IO FOR QUERY 4;
Available parameters type:
ALL # Display all overhead information
BLOCK IO # Display block IO related overhead
CONTEXT SWITCHES # Context switch related overhead
CPU # Display CPU related overhead information
IPC # Display Sending and receiving related overhead information
MEMORY # Display memory-related overhead information
PAGE FAULTS # Display page fault-related overhead information
SOURCE # Display overhead information related to Source_function, Source_file, Source_line
SWAPS # Display overhead information related to the number of swaps

4. View running threads

We can run show full processlist to view all threads running in MySQL, view their status and running time, find sites with long running times, and kill them directly.
Among them,
Id: the unique identifier of the thread. You can use the Id to kill the specified thread.
User: the user who started this thread. Ordinary accounts can only view their own threads.
Host: which IP and port initiated the connection.
db: the database operated by the thread.
Command: the thread's Command
Time: Operation duration, in seconds
State: The status of the thread
Info: The first 100 characters of the SQL statement

5. Check the server running status

Directly using the show status command to obtain more than 300 records will dazzle us, so we hope to be able to "view" some status information on demand. At this time, we can add the corresponding like clause after the show status statement. For example, if we want to view the current running time of MySQL after it is started, we can execute the following statement:
Use the SHOW STATUS command to view the status information of the server.
The specific commands are as follows: adding global represents global variables.
1) Check the number of executions of select statements:
show [global] status like 'com_select';
2) Check the number of executions of insert statements:
show [global] status like 'com_insert';
3) Check the number of executions of update statements:
show [global] ] status like 'com_update';
4) Check the number of delete statement executions:
show [global] status like 'com_delete';
5) Check the number of connections trying to connect to MySQL (whether the connection is successful or not):
show status like 'connections';
6) View the number of threads in the thread cache:
show status like 'threads_cached';
7) View the number of currently open connections:
show status like 'threads_connected';
8) View the number of threads created to handle connections:
show status like 'threads_created';
9) View the number of activated (non-sleeping) threads:
show status like 'threads_running';
10) View the number of table locks obtained immediately:
show status like 'table_locks_immediate';
11) View cannot The number of table locks acquired immediately. If the value is high and you have performance issues, you should first optimize the query and then split the table or use replication:
show status like 'table_locks_waited';
12) View the number of threads that have been created longer than slow_launch_time seconds:
show status like 'slow_launch_threads' ;
13) Check the number of queries with query time exceeding long_query_time seconds:
show status like 'slow_queries';
14) Check the running time of MySQL after this startup (unit: seconds):
show status like 'uptime';
15) Check The cost of the previous query
show status like 'last_query_cost'
16) View the query cache information
show status like 'qcache%';

6. View the running status of the storage engine
(1). SHOW ENGINE is used to display the current running information of the storage engine, including table locks and row lock information held by transactions; lock waiting status of transactions; thread semaphore waiting; file IO requests; Buffer pool statistics and other data.
(2) Query the InnoDB version
show variables like 'innodb_version'\G;
or select * from information_schema.plugins;
mysql> show variables like 'innodb_version'\G;
*************** ************ 1. row ***************************
Variable_name: innodb_version
Value: 5.7.31 -34
1 row in set (0.01 sec)
Query InnoDB status
As can be seen from the following query, it is the average number per second calculated in the last 47s, that is: the parameters change dynamically during each query.

mysql> show engine
innodb status\G;

---

** 1. row ***************

---

Type: InnoDB
Name:
Status:

===============

2022-06-27 09:33:40 0x
7fa1c8a5f700 INNODB
MONITOR OUTPUT

===============

Per second averages
calculated from the last
47 seconds

BACKGROUND THREAD
(background thread)

srv_master_thread loops
: 13285067 srv_active, 0
srv_shutdown, 1060666
6 srv_idle
srv_master_thread log
flush and writes: 238917
33
#####As shown in the above query, srv
master_thread loops is the number
of Master Thread loops
, and
a (active,
shutdown, idle) execution,
in which the increase in the number of active
is related to data changes and has nothing to do with query
.
##### It can be seen from the difference between srv
active and srv_idle
. By comparing
the values ​​​​of active and idle, Obtain
the overall load of the system. The
larger the value of active, the
busier the service.
#####srv_master_
thread log is the number of times Master
Thread writes the redo log (redo
log) to disk

SEMAPHORES (semaphore
)

OS WAIT ARRAY INFO:
reservation count 57836
764 #####OS wait array
information: expected count (
InnoDB allocation slot quota)
OS WAIT ARRAY INFO:
signal count 269276481
#####OS wait
array information: Signal count (the thread
gets the signal frequency through the array)
RW-shared spins 0,
rounds 171890114, OS
waits 34656067 ###
##RW shared lock count,
polling times, waiting time
RW-excl spins 0, rounds
1245774156, OS waits
18379769 #####
RW’s exclusive lock count,
polling times, waiting time
RW-sx spins 80621,
rounds 1707842, OS
waits 34552 #
####RW’s sx (intention exclusive
lock) count, polling Spin rounds per wait: 17
1890114.00 RW -shared, 1245774156.00 RW-



excl, 21.18 RW-sx ##
###Each spin rounds per
wait displays: Each operating
system waits for the mutex
spinlock round
...
The above statement can display various information about the current operation of the innodb storage engine. Based on this, you can find the current problems of MySQL. As long as you know that MySQL provides such a monitoring tool, you can query it in detail when needed.

1. BACKGROUND THREAD (background thread)
2. SEMAPHORES (Semaphore)
3. LATEST DETECTED DEADLOCK
4. TRANSACTIONS
5. FILE I/O
6. INSERT BUFFER AND ADAPTIVE HASH INDEX (insert buffer and adaptive hash index)
7. LOG
8. BUFFER POOL AND MEMORY (buffer pool and memory)
9. ROW OPERATIONS

7. EXPLAN execution plan

Through the slow query log, we can know which SQL statements are executed slowly, but why is it slow? Where is the slowness?
MySQL provides an execution plan query command EXPLAIN. Through this command we can view the SQL execution plan.
EXPLAIN can also be analyzed for UPDATE, DELETE and INSERT statements after MySQL 5.6.3, but usually we still use it for SELECT queries. .
SQL optimization means that there is no problem with the syntax of SQL itself, but there is a better way of writing to achieve the same purpose. for example:

使用小表驱动大表；用join改写子查询；or改成union
连接查询中，尽量减少驱动表的扇出（记录数），访问被驱动表的成本要尽量低，尽量在被驱动表的连接列上建立索引，降低访问成本；被驱动表的连接列最好是该表的主键或者是唯一二级索引列，这样被驱动表的成本会降到更低
大偏移量的limit，先过滤再排序

Let’s take a simple example for the last one. The following two statements can achieve the same purpose, but the execution efficiency of the second one is much higher than that of the first one (the storage engine uses InnoDB). Let’s feel it:
– 1. Query with large offset
mysql> SELECT * FROM user_innodb LIMIT 9000000,10;
Empty set (8.18 sec)

– 2. Filter the ID first (because the ID uses an index), then limit
mysql> SELECT * FROM user_innodb WHERE id > 9000000 LIMIT 10;
Empty set (0.02 sec)

Index Optimization
Creating appropriate indexes for slow queries is a very common and very effective method, but whether the index will be used efficiently is another topic

4. Storage engine and table structure optimization

1. Select storage engine

Under normal circumstances, we will choose InnoDB, the default storage engine of MySQL. However, when the database performance requirements are constantly improving, the choice of storage engine also becomes a key influencing factor.

For business tables with many query operations and insertion operations, it is recommended to use MyISAM; for
temporary tables, use Memory;
for businesses with large concurrency and many updates, choose InnoDB;
if you don’t know what to choose, just default.

2. Optimize fields

The final principle of field optimization is: use the smallest data type that can correctly store data.
Integer type: Different storage types have different maximum storage ranges, and naturally occupy different storage spaces
. Character type: If you are not sure about the length of the field, it must be Choose varchar, but varchar requires additional space to record the length currently occupied by the field; therefore, if the length of the field is fixed, try to use char, which will save you a lot of memory space.
Non-null: Try to set non-null fields to NOT NULL and provide default values, or use special values ​​instead of NULL.
Do not use foreign keys, triggers and view functions.
Image, audio, and video storage: Do not store large files directly; Access address of large files
Large field splitting and data redundancy

5. Business Optimization

1. Business optimization is no longer a means of MySQL tuning, but business optimization can very effectively reduce database access pressure. A typical example in this regard is Taobao 1. In the past, shopping started on the night of Double
11 model, in recent years the Double 11 pre-sales front has become longer and longer, starting more than half a month in advance, and various deposit red envelope models have emerged in endlessly. This method is called pre-sale diversion. This can divert customers' service requests, instead of waiting until the early morning of Double Eleven to place orders collectively;
2. During Double Eleven, Alipay strongly recommends using Huabei payment instead of bank card payment, although part of the consideration is to improve the stickiness of the software , but on the other hand, using Alibaba's internal server actually used by Yu'ebao has fast access speed, while using a bank card requires calling the bank interface, which is much slower in comparison.