Mysql performance tuning (eight)

Preface

1. Application optimization

In the previous articles, I have introduced a lot of measures for database optimization, including optimization of indexes and SQL optimization. However, in the actual production environment, due to the performance limitations of the database itself, some optimizations must be made to the foreground applications to reduce the access pressure of the database.

1.1 Use connection pool

For accessing the database, the cost of establishing a connection is more expensive, because we frequently create and close connections, which is more resource intensive. It is necessary for us to establish a database connection pool to improve access performance.

1.2 Reduce access to MySQL

1.2.1 Avoid repeated retrieval of data

When writing application code, you need to be able to clarify the access logic to the database. If you can get the result in one connection, you don't need to connect twice, which can greatly reduce the useless repeated requests to the database. Let’s say 需要获取书籍的id和name字段, the query code is as follows:

select id, name from tb_book;

After that, if there is a need to obtain book status information in the business logic, the query is as follows:

select id, status from tb_book;

In this way, you need to submit two requests to the database, and the database needs to perform two query operations. In fact, you can use a SQL statement to get the desired result. The implementation is as follows:

select id, name, status from tb_book;

1.2.2 Increase the cache layer

In the application, we can increase the buffer layer in the application to achieve the purpose of reducing the burden on the database. There are many buffer layers, and there are many ways to implement them, as long as they can reduce the burden on the database and meet the application requirements. Therefore, part of the data can be extracted from the database and stored on the application side as text, or use the primary cache/secondary cache provided by the framework (Mybatis, Hibernate), or use the redis database to cache data.

1.3 Load balancing

Load balancing is a very common optimization method used in applications. Its mechanism is to use a certain balancing algorithm to distribute a fixed amount of load to different servers, so as to reduce the load of a single server to achieve optimization effect.

1.3.1 Use MySQL to replicate and split queries

Through MySQL's master-slave replication, read and write separation is realized, so that addition, deletion, modification, and query operations go to the master node, and query operations go to the slave node, which can reduce the read and write capabilities of a single server. The specific architecture diagram is shown in the figure:

1.3.2 Adopt a distributed database architecture

The distributed database architecture is suitable for large data volume and high load conditions, and it has good scalability and high availability. By distributing data among multiple servers, load balance among multiple servers can be realized and service efficiency can be improved.

2. Query cache optimization in MySQL

Turn on the query cache of mysql. When the same SQL statement is executed, the server will directly read the results from the cache. When the data is modified, the previous cache will become invalid. Tables that are frequently modified are not suitable for query cache . The specific optimization process is as follows:

1. The client sends a query to the server;

2. The server sends a query, and if it hits the cache, it immediately returns the result stored in the cache. Otherwise, go to the next stage.

3. The server side performs SQL analysis and preprocessing, and then the optimizer generates the corresponding execution plan

4. MySQL calls the storage engine's API to execute the query according to the execution plan generated by the optimizer

5. Return the result to the client.

2.1 Query cache configuration

1. Check whether the current MySQL database supports query caching, the specific code is as follows:

show variables like 'have_query_cache';

The effect of execution is shown in the figure:

2. Check whether the query cache is currently enabled in MySQL

show variables like 'query_cache_type';

The effect of execution is shown in the figure:

3. Check the occupied size of the query cache

show variables like 'query_cache_size';

The effect of execution is shown in the figure:

4. View the state variables of the query cache

show status like 'Qcache%';

The effect of the execution is shown in the figure:

we can see a lot of professional terms from the above execution effect, and then I will introduce you one by one:

Qcache_free_blocks: Query the number of free memory blocks in the cache

Qcache_free_memory: Query the amount of free memory in the cache

Qcache_hits: Query cache hits

Qcache_inserts: the number of queries added to the query cache

Qcache_lowmen_prunes: The number of queries deleted from the query cache due to insufficient memory

Qcache_not_cached: The number of non-cached queries (cannot be cached or not cached due to the query_cache_type setting)

Qcache_queries_in_cache: The number of queries registered in the query cache

Qcache_total_blocks: the total number of blocks in the query cache

2.2 Turn on query cache

MySQL query cache is turned off by default, and you need to manually configure parameters query_cache_typeto enable query cache. query_cache_typeThere are three possible values for this parameter, as follows:

In the /usr/my.cnf configuration, add the following configuration:

query_cache_type=1;

The specific execution effect is as follows:

After the configuration is completed, restart to take effect; then you can execute the SQL statement on the command line for verification, execute a relatively time-consuming SQL statement, and then execute it a few more times to check the execution time of the next few times; Obtain the cache hit count of the query cache to determine whether to go to the query cache.

2.3 Query cache select option

Two options related to query caching can be specified in the select statement:

SQL_CACHE: If the query result is cacheable, and query_cache_typethe value of the system variable is ON or DEMAND, the query result is cached.

SQL_NO_CACHE: The server does not use query cache. It neither checks the query cache, nor does it check whether the result is cached, nor does it cache the query result.

select SQL_CACHE id, name from customer;
select SQL_NO_CACHE, id, name from customer;

2.4 Query cache invalidation

1. In the case of SQL inconsistency, in order to hit the query cache, the query SQL statements must be consistent.

SQL1:select count(*) from tb_item;
SQL2:select count(*) from tb_item;

2. When there are some uncertainties in the query statement, it will not be cached. For example: now(), current_date(), curdate(), rand(), uuid(), user(), database();

SQL1:select count(*) from tb_item where updatatime < now() limit 1;
SQL2：select user();
SQL3：select database();

3. Do not use any table query statements

select 'A';

4, query mysql, information_schemaor performance_schemadatabase table, query cache will not go

select * from information_schema;

5. Execute the query in the body of the stored function, trigger or event.
6. If the table changes, all cache queries that use the table will become invalid and deleted from the cache. This includes using MERGEqueries mapped to changed tables. A table can be many types of statements, such as is changed INSERT, UPDATE, DELETE, TRUNCATE TABLE, ALTER TABLE, DROP TABLEor DROP DATABASE.

Three, MySQL memory management and optimization

3.1 Principles of Memory Optimization

1. Allocate as much memory as possible to MySQL for caching, but reserve enough memory for the operating system and other programs.
2. The data file reading of the MyISAM storage engine depends on the operating system's own IO cache. Therefore, if there is a MyIsam table, more memory must be reserved for the operating system as an IO cache.
3. Cache areas such as sorting area and connection area are allocated to each database session (session). The default value setting should be reasonably allocated according to the maximum number of connections. If the setting is too large, it will not only waste resources, but also consume more concurrent connections. High will cause physical memory to run out.

3.2 MyISAM memory optimization

The MyISAM storage engine uses key_buffercached index blocks to accelerate the read and write speed of MyISAM indexes. For the data block of the MyISAM table. MySQL does not have a special caching mechanism and is completely dependent on the IO cache of the operating system.

key_buffer_size
key_buffer_size determines the size of the MyISAM index block cache, which directly affects the access efficiency of MyISAM tables. You can set the value of key_buffer_size in the MySQL parameter file. For a general MyISAM database, it is recommended to allocate at least 1/4 of the available memory to key_buffer_size. We /usr/my.cnfmake the following configuration in:

key_buffer_size=512M;

If read_buffer_size needs to scan MyISAM tables frequently, you can increase the value of read_buffer_size to improve performance. But it should be noted that read_buffer_size is sessionexclusive to each , if the default value is set too large, it will cause a waste of memory.
read_rnd_buffer_size
For queries of MyISAM tables that need to be sorted, if order bySQL with clauses is added, the value of read_rnd_buffer_size can be appropriately increased to improve the performance of this type of SQL. But it should be noted that read_rnd_buffer_size is sessionexclusive to each , if the default value is set too large, it will cause a waste of memory.

3.3 InnoDB memory optimization

InnoDB uses a memory area as an IO buffer pool, which is not used to cache InnoDB index blocks, but is also used to cache InnoDB data blocks.

innodb_buffer_pool_size
This variable determines the maximum buffer size of InnoDB storage engine table data and index data. In the case of ensuring that the operating system and other programs have sufficient memory available, the larger the value of innodb_buffer_pool_size, the higher the cache hit rate, the less disk I/O required to access InnoDB tables, and the higher the performance.

innodb_buffer_pool_size=512M;

innodb_log_buffer_size
determines the size of InnoDB's important log buffer. For large transactions that may generate a large number of updated records, increasing the size of innodb_log_buffer_size can prevent Innodb from performing unnecessary log write operations to disk before the transaction is committed.

innodb_log_buffer_size=10M;

Fourth, MySQL concurrency parameter adjustment

In fact, MySQL Server is a multi-threaded structure, including background threads and customer service threads. Multithreading can effectively use server resources and improve the concurrent performance of the database. In MySQL, the thread control concurrent connections and the main parameters including max_connections, back_log, thread_cache_size, table_open_cache.
Use max_connections to control the maximum number of connections allowed to the MySQL database. The default value is 151. If the state variable is connection_errors_max_connections0 and keeps increasing, it means that there are continuous connection requests failing because the number of database connections has reached the maximum allowed value. This can be considered to increase max_connectionsthe value.
The maximum number of connections that MySQL may support depends on many factors, including the quality of the thread library of a given operating system platform, the size of the memory, the load of each connection, the CPU processing speed, and the expected response time. Under the Linux platform, it is not difficult for a server with good performance to support 500-1000 connections. It needs to be evaluated and set according to the server performance.
The back_log parameter controls the size of the backlog request stack set when MySQL listens on the TCP port. If the number of MySQL connections reaches max_connections, new requests will be stored in the stack to wait for a connection to release resources. The number of the stack is back_log. If the number of waiting connections exceeds back_log, connection resources will not be granted. An error will be reported. The default value before version 5.6.6 is 50, and the default value for later versions is 50+(max_connections/5), but the maximum is not more than 900. If you need the database to process a large number of connection requests in a relatively short period of time, you can consider increasing the value of back_log appropriately.
This parameter is used to control the number of table caches that all SQL statement execution threads can open. When executing SQL statements, each SQL execution thread must open at least one table cache. The value of this parameter should be set according to the maximum number of connections max_connectionsand the maximum number of tables involved in the execution of the associated query for each connection:

max connections x N;

In order to speed up the connection to the database, a certain number of customer service threads in MySQL or cache have been prepared for reuse. The parameter thread_cache_size can control the number of MySQL cache customer service threads.
This parameter is used to set the time for InnoDB transactions to wait for row locks. The default value is 50ms, which can be dynamically set as needed. For business systems that require fast feedback, the waiting time of row locks can be adjusted to avoid prolonged suspension of transactions; for batch processing programs running in the background, the waiting time of row locks can be adjusted to avoid A large rollback operation occurred.

to sum up

In the previous article, we introduced SQL optimization , including optimizing insert statements, optimizing order by statements, filesort optimization, optimizing group by statements, optimizing nested queries, optimizing or conditions, optimizing paging queries, and using SQL hints. This article introduces you to application optimization, query cache optimization in mysql, memory management optimization in mysql, and concurrency parameter adjustment in mysql. Next, we first introduce you to application optimization. Therefore, mysql is a very important skill. Almost every job in the computer needs a mysq skill. Therefore, we need special mastery. Life is endless and struggle is endless. We work hard every day, study hard, constantly improve our abilities, and believe that we will learn something. Come on! ! ! Finally, because of participating in an event, this is my exclusive link https://lbs.qq.com?lbs_invite=HZZRFLH.