Article Directory
Configuration optimization
MySQL parameter optimization is related to different websites, and their online volume, visit volume, number of posts, network conditions, and machine hardware configuration. Optimization cannot be completed at one time. It requires constant observation and debugging to get the best. effect. The main variables that have a greater impact on performance optimization are listed below, which are mainly divided into connection request variables and buffer variables.
Connection request variables
1.max_connections
MySQL's maximum number of connections. If the number of concurrent connection requests from the server is relatively large, it is recommended to increase this value to increase the number of parallel connections. Of course, this is based on the situation that the machine can support, because if the number of connections is greater, MySQL A connection buffer will be provided for each connection, and the more memory will be spent , so it is necessary to adjust the value appropriately and not blindly increase the setting value.
If the value is too small, ERROR 1040: Too many connections error will often appear, you can pass
mysql> show status like 'connections';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Connections | 3 |
+---------------+-------+
1 row in set (0.00 sec)
The wildcard character checks the number of connections in the current state (the number of connections trying to connect to MySQL (regardless of whether the connection is successful)) to determine the value of this value.
mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 | #最大连接数
+-----------------+-------+
1 row in set (0.00 sec)
mysql> show status like 'max_used_connections';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| Max_used_connections | 3 | #响应的连接数
+----------------------+-------+
1 row in set (0.00 sec)
max_used_connections / max_connections * 100% (ideal value ≈ 85%) If max_used_connections is the same as max_connections, then max_connections is set too low or exceeds the upper limit of server load. If it is less than 10%, it is set too large.
How to set max_connections?
Method 1:
mysql> set GLOBAL max_connections=1000;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 1000 |
+-----------------+-------+
1 row in set (0.00 sec)
Method Two:
Modify the /etc/my.cnf file and add the following content under [mysqld], such as setting the maximum number of connections to 500
[root@mysqld01 ~]# cat /etc/my.cnf
[mysqld]
.....
max_connections = 500
#重启mysql服务
2.back_log
The number of connections that MySQL can temporarily store. It will work when the main MySQL thread gets a lot of connection requests in a short period of time. If the MySQL connection data reaches max_connections, new requests will be stored in the stack to wait for a connection to release resources. The number of the stack is back_log. If the number of waiting connections exceeds back_log, connection resources will not be granted. The back_log value indicates how many requests can be stored on the stack in a short period of time before MySQL temporarily stops answering new requests. Only if you expect to have many connections in a short period of time, you need to increase it.
When observing your host process list (mysql> show full processlist), you find a large number of xxxxx | unauthenticated user | xxx.xxx.xxx.xxx | NULL | Connect | NULL | login |
NULL processes to be connected, you must increase the back_log Value or increase the value of max_connections. by
mysql> show variables like 'back_log'; #查看back_log的设置
//Set the back_log value
Add the following content under [mysqld]
[root@mysqld01 ~]# tail -1 /etc/my.cnf
back_log = 150
重启mysqld
3.wait_timeout and interactive_timeout wait_timeout
wait_timeout-refers to the number of seconds that MySQL waits before closing a non-interactive connection interactive_time-refers to the number of seconds MySQL needs to wait before closing an interactive connection, such as when entering on a terminal mysql management, even if the interactive connection is used, at this time, if the time of no operation exceeds the time set by interactive_time, it will be automatically disconnected. The default value is 28800, and the adjustable value is 7200.
Impact on performance:
wait_timeout:
- If the setting is too small, then the connection will be closed quickly, so that some persistent connections will not work
- If the setting is too large, it is easy to cause the connection to open for too long. During the show processlist, you can see too many connections in the sleep state, resulting in too many connections error
- It is generally hoped that wait_timeout is as low as possible. The setting of interactive_timeout will not have much impact on your web application. Check wait_timeout and interactive_timeout.
mysql> show variables like 'wait_timeout';
mysql> show variables like 'interactive_timeout';
//Modify the wait_timeout and interactive_timeout values, add the following content under [mysqld]
[root@mysqld01 ~]# tail -2 /etc/my.cnf
wait_timeout=100
interactive_timeout=100
重启mysqld
Buffer variable
Global buffer:
1.key_buffer_size
key_buffer_size specifies the size of the index buffer, which determines the speed of index processing, especially the speed of index reading. By checking the status values Key_read_requests and Key_reads, you can know whether the key_buffer_size setting is reasonable. The ratio of key_reads
/ key_read_requests should be as low as possible, at least 1:100, 1:1000 is better (the above status value can be obtained using SHOW STATUSLIKE'key_read%').
There are a total of 6 index read requests. There are 3 requests that are not found in the memory to read the index directly from the hard disk, and calculate the probability of index misses in the cache: key_cache_miss_rate = Key_reads / Key_read_requests * 100% =50% key_buffer_size only for MyISAM tables effect. Even if you do not use MyISAM tables, but the internal temporary disk tables are MyISAM tables, you must use this value. You can use the check status value created_tmp_disk_tables to know the details.
How to adjust the key_buffer_size The default configuration value is 8388608 (8M), the host has 4GB of memory, and the adjustable value is 268435456 (256MB). Modify the /etc/my.cnf file and add the following content under [mysqld]
[root@mysqld01 ~]# tail -1 /etc/my.cnf
key_buffer_size=268435456
//或key_buffer_size=256M
重启mysqld
2. query_cache_size
query_cache_size (QC for short for query cache) Using query buffer, MySQL stores the query results in the buffer. In the future, for the same SELECT statement (case sensitive), the results will be read directly from the buffer. If a SQL query starts with select, the MySQL server will try to use the query cache for it.
Note: Two SQL statements, as long as the difference is even one character (for example, the case is different; one more space, etc.), then the two SQL statements will use a different CACHE.
By checking the status value'Qcache%', you can know whether the query_cache_size setting is reasonable (the above status value can be obtained using show status like'qcache%').
mysql> show status like 'qcache%';
+-------------------------+---------+
| Variable_name | Value |
+-------------------------+---------+
| Qcache_free_blocks | 1 |
| Qcache_free_memory | 1031832 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 1 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 1 |
+-------------------------+---------+
8 rows in set (0.02 sec)
Qcache_free_blocks : The number of adjacent memory blocks in the cache. If the value is displayed larger, it means that there are more memory fragments in the Query Cache, and FLUSH QUERY CACHE will defragment the fragments in the cache to obtain a free block. Note: When a table is updated, the cache blocks related to it will be freed. But this block may still exist in the queue, unless it is at the end of the queue. You can use FLUSH QUERY CACHE statement to clear free blocks
Qcache_free_memory : The current remaining memory size in Query Cache. Through this parameter, we can more accurately observe whether the Query Cache memory size in the current system is sufficient, whether it needs to be increased or is too much.
Qcache_hits : Indicates how many times the cache has been hit. We can mainly use this value to verify the effect of our query cache. The larger the number, the better the cache effect.
Qcache_inserts : Indicates how many misses and then inserts, meaning that the new SQL request is not found in the cache, and the query processing has to be executed. After the query processing is executed, the result is inserted into the query cache. The greater the number of such cases, the less application of the query cache and the unsatisfactory effect. Of course, the query cache is empty right after the system is started, which is normal.
Qcache_lowmem_prunes : How many queries were cleared out of the Query Cache due to insufficient memory. Through the combination of "Qcache_lowmem_prunes" and "Qcache_free_memory", we can understand more clearly whether the memory size of Query Cache in our system is really sufficient, and whether Query is swapped out because of insufficient memory very frequently. This number is best viewed for a long time;
if this number is increasing, it means that the fragmentation may be very serious, or the memory may be small. (The above free_blocks and free_memory can tell you which situation it belongs to)
Qcache_not_cached : The number of queries that are not suitable for caching, usually because these queries are not SELECT statements or use functions such as now().
Qcache_queries_in_cache : the number of queries in the cache in the current Query Cache;
Qcache_total_blocks : the number of blocks in the current QueryCache;
//Let's check the server configuration about query_cache
mysql> show variables like "%query_cache%";
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| have_query_cache | YES |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 1048576 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+
6 rows in set (0.00 sec)
It can be seen from the above that query_cache_type is off, which means that no interpretation of any query field is cached:
query_cache_limit: Queries over this size will not be cached
query_cache_min_res_unit: the minimum size of the cache block, the configuration of query_cache_min_res_unit is a "double-edged sword", the default is 4KB, a large setting is good for big data queries, but if your queries are all It is small data query, it is easy to cause memory fragmentation and waste.
query_cache_size: query cache size (Note: The smallest unit of QC storage is 1024 byte, so if you set a
value that is not a multiple of 1024 , this value will be rounded to the nearest value equal to a multiple of 1024.)
query_cache_type: Cache type, which determines what kind of query to cache. Note that this value cannot be set casually. It must be set to a number. The optional items and descriptions are as follows:
uery_cache_type三个参数的含义:
query_cache_type=0（OFF）关闭
query_cache_type=1（ON）缓存所有结果，除非select语句使用SQL_NO_CACHE禁用查询缓存
query_cache_type=2(DEMAND)，只缓存select语句中通过SQL_CACHE指定需要缓存的查询
If it is set to 0, it can be said that your cache is useless at all, which is equivalent to disabled. If set to 1, all results will be cached, unless your select statement uses SQL_NO_CACHE to disable query caching. If it is set to 2, only the queries that need to be cached are specified in the select statement through SQL_CACHE.
query_cache_wlock_invalidate: When other clients are writing to the MyISAM table, if the query is in the query cache, whether to return the cache result or wait for the write operation to complete before reading the table to obtain the result.
Query cache fragmentation rate = Qcache_free_blocks / Qcache_total_blocks * 100% If the query cache fragmentation rate exceeds 20%, you can use FLUSH QUERY CACHE to defragment the cache fragments, or try to reduce query_cache_min_res_unit, if your queries are all small data volumes. Query cache utilization = (query_cache_size – Qcache_free_memory) / query_cache_size * 100% If the query cache utilization is below 25%, it means that the query_cache_size setting is too large and can be appropriately reduced; if the query cache utilization is above 80% and Qcache_lowmem_prunes> 50, it means The query_cache_size may be a bit small, or there is too much fragmentation. Query cache hit rate = Qcache_hits/(Qcache_hits+Qcache_inserts) * 100%
Limitations of Query Cache: a) External query SQL in all subqueries cannot be cached; b) Query in Procedure, Function and Trigger cannot be cached; c) Query containing many other functions that may get different results each time they are executed Cannot be Cached. In view of the above limitations, in the process of using Query Cache, it is recommended to use it through precise settings. Only the data of the appropriate table can enter the Query Cache, and only certain query results can be cached.
//Set query_cache_size
[root@mysqld01 ~]# tail -2 /etc/my.cnf
query_cache_size=256M
query_cache_type=1
//重启mysqld
3.max_connect_errors
max_connect_errors is a security-related counter value in MySQL. It is responsible for preventing too many failed clients to prevent brute force cracking of passwords. When the specified number of times is exceeded, the MYSQL server will prohibit the host's connection request until the mysql server Restart or clear the relevant information of this host through the flush hosts command. The value of max_connect_errors has nothing to do with performance.
Modify the /etc/my.cnf file and add the following content under [mysqld] max_connect_errors=20 After restarting MySQL Server and entering, check that the settings have taken effect.
max_connect_errors=20
or
mysql> set GLOBAL max_connect_errors=20;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like 'max_connect_errors';
+--------------------+-------+
| Variable_name | Value |
+--------------------+-------+
| max_connect_errors | 20 |
+--------------------+-------+
1 row in set (0.00 sec)
4.sort_buffer_size
sort_buffer_size Each thread that needs to sort allocates a buffer of this size. Increasing this value speeds up ORDER BY or GROUPBY operations. Sort_Buffer_Size is a connection-level parameter. When each connection (session) needs to use this buffer for the first time, the set memory is allocated at one time. Sort_Buffer_Size is not as big as possible, because it is a connection-level parameter, too large setting + high concurrency may exhaust system memory resources.
For example: 500 connections will consume 500*sort_buffer_size(2M)=1G memory
//Set sort_buffer_size Modify the /etc/my.cnf file, add the following content under [mysqld] sort_buffer_size = 2M After restarting MySQL Server, check that the setting has taken effect.
5.max_allowed_packet
max_allowed_packet = 32M MySQL will limit the packet size accepted by the Server according to the configuration file. Sometimes large inserts and updates are limited by the max_allowed_packet parameter, causing write or update failures. The maximum value is 1GB, and a multiple of 1024 must be set.
6.join_buffer_size
join_buffer_size = 2M The size of the associated cache between tables. Like sort_buffer_size, the allocated memory corresponding to this parameter is also exclusive to each connection.
7.thread_cache_size
thread_cache_size = 300 server thread cache, this value indicates that the number of threads stored in the cache can be reused. When the connection is disconnected, then the client thread will be placed in the cache to respond to the next client instead of being destroyed (premise The number of caches has not reached the upper limit). If the thread is requested again, the request will be read from the cache. If the cache is empty or a new request, then the thread will be recreated. If there are many new threads, Increasing this value can improve system performance. By comparing Connections and Threads_created state variables, you can see the effect of this variable.
The setting rules are as follows: 1GB is configured as 8, 2GB is configured as 16, and 3GB is configured as 32, 4GB or higher memory can be configured larger. The server's thread processing this client will be cached to respond to the next client instead of being destroyed (provided that the number of caches does not reach the upper limit)
The number of connections attempted to connect to MySQL (regardless of whether the connection was successful or not)
Threads_cached: Represents how many idle threads are currently in the thread cache at this moment.
Threads_connected: Represents the number of currently established connections. Because a connection requires one thread, it can also be regarded as the number of threads currently being used.
Threads_created: Represents the number of threads that have been created since the last service startup. If the Threads_created value is found to be too large, it means that the MySQL server has been creating threads, which is also resource-intensive. You can appropriately increase the thread_cache_size value in the configuration file.
Threads_running: Represents the number of threads currently active (non-sleeping). It does not mean the number of threads in use. Sometimes the connection is established, but the connection is in sleep state.
Configure several variables of InnoDB
1.innodb_buffer_pool_size
innodb_buffer_pool_size For InnoDB tables, innodb_buffer_pool_size has the same effect as key_buffer_size for MyISAM tables. InnoDB uses this parameter to specify the size of memory to buffer data and indexes. For a single MySQL database server, the maximum value can be set to 80% of the physical memory. According to the MySQL manual, for a machine with 2G memory, the recommended value is 1G (50%). If your data volume is not large and will not increase sharply, then there is no need to set innodb_buffer_pool_size too large.
mysql> show variables like 'innodb_buffer_pool_size';
+-------------------------+-----------+
| Variable_name | Value |
+-------------------------+-----------+
| innodb_buffer_pool_size | 134217728 |
+-------------------------+-----------+
1 row in set (0.00 sec)
//Set innodb_buffer_pool_size to
modify the /etc/my.cnf file, add the following content under [mysqld]
[root@mysqld01 ~]# tail -1 /etc/my.cnf
innodb_buffer_pool_size = 2048M
2.
innodb_flush_log_at_trx_commit innodb_flush_log_at_trx_commit mainly controls the time point when innodb writes the data in the log buffer into the log file and flushes the disk. The values are 0, 1, and 2 respectively.
0, means that when the transaction is committed, no log write operation is performed, but the data in the log buffer is written to the log file every second and the disk is flushed; 1, then every second or every transaction commit will be The operation that causes log file writing and flush disk ensures the ACID of the transaction; set to 2, each transaction commit causes the action of writing the log file, but the flush disk operation is completed every second. The actual test found that this value has a great influence on the speed of inserting data. When it is set to 2, it only takes 2 seconds to insert 10000 records, when it is set to 0, it only takes 1 second, and when it is set to 1, it takes 229 seconds. Therefore, the MySQL manual also recommends combining insert operations into one transaction as much as possible, which can greatly increase the speed. According to the MySQL manual, this value can be set to 0 or 2 under the premise of allowing the risk of losing the most recent transaction.
3. innodb_thread_concurrency
innodb_thread_concurrency = 0 This parameter is used to set the number of concurrent innodb threads, the default value is 0 means no limit, if you want to set it is the same as the number of CPU cores of the server or twice the number of cores of the cpu, it is recommended to use the default setting , Generally 8.
mysql> show variables like 'innodb_thread_concurrency';
+---------------------------+-------+
| Variable_name | Value |
+---------------------------+-------+
| innodb_thread_concurrency | 0 |
+---------------------------+-------+
1 row in set (0.00 sec)
mysql> set GLOBAL innodb_thread_concurrency=8;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like 'innodb_thread_concurrency';
+---------------------------+-------+
| Variable_name | Value |
+---------------------------+-------+
| innodb_thread_concurrency | 8 |
+---------------------------+-------+
1 row in set (0.01 sec)
4.
innodb_log_buffer_size innodb_log_buffer_size This parameter determines the memory size used by some log files, with M as the unit. Larger buffers can improve performance, and for larger transactions, the cache size can be increased. innodb_log_buffer_size=32M
5. innodb_log_file_size
innodb_log_file_size = 50M This parameter determines the size of the data log file, in M, a larger setting can improve performance.
7. innodb_log_files_in_group
innodb_log_files_in_group = 3 To improve performance, MySQL can write log files to multiple files in a circular manner. Recommended setting is 3
8. read_buffer_size
read_buffer_size = 1M MySql read buffer size. The request for sequential scan of the table will allocate a read buffer, and MySql will allocate a memory buffer for it. If the sequential scan requests for the table are very frequent, and you think that frequent scans are too slow, you can improve its performance by increasing the value of this variable and the size of the memory buffer. Like sort_buffer_size, the allocated memory corresponding to this parameter is also exclusive to each connection.
9. read_rnd_buffer_size
read_rnd_buffer_size = 16M MySql random read (query operation) buffer size. When the rows are read in any order (for example, in the sort order), a random read buffer will be allocated. When sorting and querying, MySql will scan the buffer first to avoid disk search and increase the query speed. If you need to sort a large amount of data, you can increase the value appropriately. But MySql will issue this buffer space for each client connection, so this value should be set appropriately as much as possible to avoid excessive memory overhead. Note: Sequential reading means that the required row data can be read sequentially according to the leaf node data of the index. Random read refers to the general need to find the actual row data according to the primary key in the secondary index leaf node, and the secondary index and the primary key are located in different data segments, so the access method is random.
10. bulk_insert_buffer_size
bulk_insert_buffer_size= 64M batch insert data buffer size, which can effectively improve the insertion efficiency, the default is 8M
11. binary log
log-bin=/usr/local/mysql/data/mysql-bin binlog_cache_size = 2M //The memory allocated for each session is used to store the cache of the binary log during the transaction and improve the record bin-log s efficiency. There is no big transaction. If the dml is not very frequent, you can set it to be smaller. If the transaction is large and many, and the dml operation is frequent, you can adjust it to a larger value. The former suggestion is –1M, the latter suggestion is: 2–4M max_binlog_cache_size = 8M //Indicates the maximum cache memory size that binlog can use max_binlog_size = 512M //Specify the size of the binlog log file, if the current log size reaches max_binlog_size , And a new binary log will be created automatically. You cannot set this variable to be larger than 1GB or smaller than 4096 bytes. The default value is 1GB. When importing large-capacity sql files, it is recommended to close sql_log_bin, otherwise the hard disk cannot hold it, and it is recommended to delete them regularly. expire_logs_days = 7 //Defines the time for mysql to clear expired logs. The number of days that the binary log is automatically deleted. The default value is 0, which means "no automatic deletion". mysqladmin flush-logs can also restart a new binary log.
12.log_queries_not_using_indexes
log_queries_not_using_indexes Turn on this option to truly record the query that returns all rows.
Comparison between before and after optimization
//Before optimization
[root@mysqld01 ~]# mysqlslap --defaults-file=/etc/my.cnf --concurrency=10 --iterations=1 --create-schema='test1' --query='select * from test1.tb1' --engine=innodb --number-of-queries=2000 -uroot -p123.com –verbose
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
Running for engine innodb
Average number of seconds to run all queries: 0.101 seconds
Minimum number of seconds to run all queries: 0.101 seconds
Maximum number of seconds to run all queries: 0.101 seconds
Number of clients running queries: 10
Average number of queries per client: 200
//Optimized
[root@mysqld01 ~]# mysqlslap --defaults-file=/etc/my.cnf --concurrency=10 --iterations=1 --create-schema='test1' --query='select * from test1.tb1' --engine=innodb --number-of-queries=2000 -uroot -p123.com –verbose
mysqlslap: [Warning] Using a password on the command line interface can be insecure.
Benchmark
Running for engine innodb
Average number of seconds to run all queries: 0.039 seconds
Minimum number of seconds to run all queries: 0.039 seconds
Maximum number of seconds to run all queries: 0.039 seconds
Number of clients running queries: 10
Average number of queries per client: 200
//Optimize parameters
[root@mysqld01 ~]# cat /etc/my.cnf
[mysqld]
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
port = 3306
server_id = 1
socket = /usr/local/mysql/mysql.sock
log-error = /usr/local/mysql/data/mysqld.err
pid-file = /usr/local/mysql/data/mysql.pid
slow_query_log = 1
slow_query_log_file = /usr/local/mysql/data/slow-query.log
long_query_time = 1
log-queries-not-using-indexes
max_connections = 1024
back_log = 128
wait_timeout = 60
interactive_timeout = 7200
key_buffer_size=256M
query_cache_size = 256M
query_cache_type=1
query_cache_limit=50M
max_connect_errors=20
sort_buffer_size = 2M
max_allowed_packet=32M
join_buffer_size=2M
thread_cache_size=200
innodb_buffer_pool_size = 2048M
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size=32M
innodb_log_file_size=128M
innodb_log_files_in_group=3
log-bin=mysql-bin
binlog_cache_size=2M
max_binlog_cache_size=8M
max_binlog_size=512M
expire_logs_days=7
read_buffer_size=1M
read_rnd_buffer_size=16M
bulk_insert_buffer_size=64M
log-error = /usr/local/mysql/data/mysqld.err