RDS MySQL Parameter Tuning Best Practices

Little Black Sugar 2017-12-01 13:07:49 Views 132 Comments 0
mysql innodb RDS performance thread database Cache data security tokudb

Summary: Preface Many times, RDS users often ask how to tune the parameters of RDS MySQL. In order to answer this question, Write a blog to explain: Which parameters cannot be modified, and which parameters can be modified; Are these modified parameters already the best settings, and how to make good use of these parameters; Which parameters can be changed Careful users are buying RDS You will see that the maximum number of connections and memory that can be provided by different specifications are different, so the limiting parameters of these product specifications: the number of connections and memory users cannot modify, if there is a bottleneck in the memory or the number of connections: memory Bottleneck: OOM occurs in the instance, which will lead to switching between active and standby connections. Bottleneck: If the application cannot establish a new connection to the database, application optimization, slow SQL optimization, or flexible upgrade of instance specifications are required to solve the problem.

Preface
Many times, RDS users often ask how to tune the parameters of RDS MySQL. In order to answer this question, I wrote a blog to explain:

which parameters cannot be modified, and which parameters can be modified;
are these parameters provided for modification already It is the best setting, how can we make good use of these parameters;
which parameters can be changed?
Careful users will see when purchasing RDS that the maximum number of connections and memory that different specifications can provide are different, so these product specifications limit parameters : The number of connections and memory cannot be modified by the user. If there is a bottleneck in the memory or the number of connections:

Memory bottleneck: OOM occurs in the instance, which will lead to the switch between the master and the backup.
Connection bottleneck: The application cannot establish a new connection to the database
. Optimization, slow SQL optimization, or flexible upgrade of instance specifications to solve the problem.

There are also some parameters related to the security of primary and secondary data, such as innodb_flush_log_at_trx_commit, sync_binlog, gtid_mode, semi_sync, binlog_format, etc. In order to ensure the security of primary and secondary data, they are not currently available for users to modify.

In addition to the above parameters, most of the parameters have been optimized by the DBA team and the source code team. Users do not need to adjust the online parameters too much to run the database well. However, these parameters are only suitable for most application scenarios, and individual special scenarios still need to be treated individually. For example, if the tokudb engine is used, at this time, the memory ratio (tokudb_buffer_pool_ratio) that can be used by the tokudb engine needs to be adjusted; If there is a large lock timeout, then you need to adjust the size of the innodb_lock_wait_timeout parameter to suit the application and so on.

How to Adjust Parameters
Below I will introduce some of the more important parameters that can be modified in the console. If these parameters are not set properly, performance problems or application errors may occur.

The role of open_files_limit
: This parameter is used to control the number of file handles that the MySQL instance can open and use at the same time.
Reason: When more and more tables in the database (MyISAM engine tables need to consume file descriptors when they are accessed, the InnoDB engine will manage the tables that have been opened - table_open_cache) open more and more, the files allocated to each instance will be consumed. The number of handles. The open_files_limit set by RDS when the instance is initially initialized is 8192. When the number of open tables exceeds this parameter, all database requests will report errors.
Phenomenon: If the parameter setting is too small, the application may report an error
[ERROR] /mysqld: Can't open file: './mysql/user.frm' (errno: 24 -Too many open files);
Suggestion: increase the value of open_files_limit, RDS can currently support a maximum of 65535, and suggest Replace the MyISAM storage engine with the InnoDB engine.

The function of back_log
: MySQL will create a new thread corresponding to it every time it processes a connection request. During the creation of a new thread by the main thread, if the front-end application has a large number of short connection requests reaching the database, MySQL will limit the new connection at this moment. Enter the request queue, which is controlled by the parameter back_log. If the number of waiting connections exceeds back_log, new connection requests will not be accepted, so if MySQL needs to be able to handle a large number of short connections, you need to increase the size of this parameter.
Phenomenon: If the parameter is too small, the application may report an error
SQLSTATE[HY000] [2002] Connection timed out;
Suggestion: increase the value of this parameter, pay attention to the need to restart the instance, the default value of the initialized value of RDS is 50, now initialize The value has been increased by 3000.

Function of innodb_autoinc_lock_mode
: After MySQL 5.1.22, InnoDB introduced the parameter innodb_autoinc_lock_mode to solve the problem of self-incrementing the primary key lock table, which is used to control the locking mechanism of the self-incrementing primary key. The value of this parameter can be set to 0/1/2, and the default value of RDS The parameter value of 1 indicates that InnoDB uses lightweight mutex locks to obtain self-increment locks instead of the most primitive table-level locks, but in load data (including: INSERT … SELECT, REPLACE … SELECT) scenarios will use self-incrementing locks. Table locks, which may cause the application to deadlock when importing data concurrently.
Phenomenon: If the application uses load data (including: INSERT … SELECT, REPLACE … SELECT) to import data concurrently, a deadlock occurs:
RECORD LOCKS space id xx page no xx n bits xx index PRIMARY of table xx.xx trx id xxx lock_mode X insert intention waiting. TABLE LOCK table xxx.xxx trx id xxxx lock mode AUTO-INC waiting;
suggestion: it is recommended to change the parameter setting to 2, which means that in all cases, a lightweight mutex lock is used for insertion (only for row mode), In this way, the deadlock of auto_inc can be avoided, and the performance will be greatly improved in the scenario of INSERT ... SELECT (note that this parameter is set to 2, and the format of binlog needs to be set to row).

The function of query_cache_size
: This parameter is used to control the memory size of the MySQL query cache; if MySQL opens the query cache, it will lock the query cache first when executing each query, and then judge whether it exists in the query cache. If it exists, the result will be returned directly. If it does not exist, then perform engine query and other operations; at the same time, operations such as insert, update and delete will invalidate the query cache. This failure also includes any changes in the structure or index. The maintenance cost of cache failure is high, which will give MySQL to MySQL. It brings a lot of pressure, so when our database is not updated so frequently, query cache is a good thing, but if, conversely, writes are very frequent and concentrated on a few tables, then query cache The lock mechanism of lock will cause frequent lock conflicts. The write and read of this table will wait for each other to unlock the query cache lock, which will reduce the query efficiency of select.
Phenomenon: There are a large number of connections in the database with the status of checking query cache for query, Waiting for query cache lock, and storing result in query cache;
Suggestion: RDS disables the query cache function by default. After the above situation, you can close the query cache; of course, you can also open the query cache in some cases, such as: using the query cache skillfully to solve database performance problems.

The role of net_write_timeout
: the timeout for waiting for a block to be sent to the client.
Phenomenon: If the parameter setting is too small, the client may report the error the last packet successfully received from the server was milliseconds ago, the last packet sent successfully to the server was milliseconds ago.
Recommendation: This parameter is set to 60S by default in RDS. Generally, when the network conditions are relatively poor, or when the client takes a long time to process each block, the connection interruption caused by the setting of net_write_timeout is too small. It is recommended to increase this parameter. The size of the parameter; the role of

tmp_table_size
: This parameter is used to determine the maximum value of the internal memory temporary table, which is allocated by each thread (the actual limit is the minimum value of tmp_table_size and max_heap_table_size), if the memory temporary table exceeds the limit, MySQL It will be automatically converted into a disk-based MyISAM table. When optimizing query statements, avoid using temporary tables. If it is unavoidable, ensure that these temporary tables exist in memory.
Phenomenon: If a complex SQL statement contains group by/distinct, etc., which cannot be optimized by indexes, but a temporary table is used, the SQL execution time will be prolonged.
Recommendation: If there are many group by/distinct statements in the application, and the database has enough memory, you can increase the value of tmp_table_size (max_heap_table_size) to improve query performance.

New parameters of RDS MySQL
The following introduces several useful new parameters of RDS MySQL.

rds_max_tmp_disk_space
function: used to control the size of temporary files that MySQL can use. The initial default value of RDS is 10G. If the temporary file exceeds this size, the application will report an error.
Phenomenon: The table '/home/mysql/dataxxx/tmp/#sql_2db3_1' is full.
Suggestion: It is necessary to analyze whether the SQL statement that causes the increase of temporary files can be optimized by indexing or other methods. Secondly, if it is determined that the space of the instance is sufficient, you can increase the value of this parameter to ensure that the SQL can be executed normally. Note that this parameter requires restarting the instance;

tokudb_buffer_pool_ratio
is used to control the buffer memory size that the TokuDB engine can use. For example, if innodb_buffer_pool_size is set to 1000M, and tokudb_buffer_pool_ratio is set to 50 (representing 50%), then the buffer memory size that can be used by the tables of the tokudb engine is It is 500M;
Suggestion: This parameter is set to 0 by default in RDS. If the tokudb engine is used in RDS, it is recommended to increase this parameter to improve the access performance of the TokuDB engine table. Adjusting this parameter requires restarting the database instance.

The role of max_statement_time
: used to control the maximum execution time of the query in MySQL. If the time set by this parameter is exceeded, the query will automatically fail. The default is unlimited.
Suggestion: If the user wants to control the execution time of SQL in the database, this parameter can be enabled, and the unit is milliseconds.
Phenomenon: ERROR 3006 (HY000): Query execution was interrupted, max_statement_time exceeded

rds_threads_running_high_watermark
Function: It is used to control the number of concurrent queries in MySQL. For example, if the value of rds_threads_running_high_watermark is set to 100, the number of concurrent queries allowed by MySQL is 100 at the same time, exceeding the water mark The query will be rejected, this parameter is used in conjunction with rds_threads_running_ctl_mode (default is select).
Suggestion: This parameter is often used in spike or large concurrency scenarios, and has a better protection effect on the database.

Copyright statement: The content of this article is contributed by Internet users voluntarily, and this community does not own the ownership and does not assume relevant legal responsibilities. If you find any content suspected of plagiarism in this community, please send an email to: [email protected] to report and provide relevant evidence. Once verified, this community will immediately delete the allegedly infringing content.