Other database tuning strategies
1. Database tuning measures
1.1 Tuning goals
- Save system resources as much as possible so that the system can provide greater load services. (greater throughput)
- Reasonable structural design and parameter adjustment to improve the speed of user operation response. (faster response)
- Reduce system bottlenecks and improve the overall performance of the MySQL database.
1.2 How to locate tuning problems
- User feedback (main)
- Log analysis (main)
- Server resource usage monitoring
- Database internal status monitoring
- other
1.3 Dimensions and steps of tuning
Step 1: Choose the right DBMS
Step 2: Optimize table design
Step 3: Optimize Logical Query
Step 4: Optimize physical queries
Physical query optimization is to use physical optimization technology (such as indexing, etc.) after determining the logical query optimization, and estimate various possible access paths by calculating the cost model, so as to find the execution plan with the lowest cost among the execution methods.
Step 5: Use Redis or Memcached as cache
Because the data is stored in the database, we need to retrieve the data from the database layer and put it into the memory for business logic operations. When the number of users increases, frequent data queries will consume a lot of database resources. If we put commonly used data directly into memory, the query efficiency will be greatly improved.
Key-value storage database can help us solve this problem.
Commonly used key-value storage databases include Redis and Memcached, both of which can store data in memory.
Step 6: Library-Level Optimization
1. Separation of reading and writing
2. Data fragmentation
2. Optimize MySQL server
2.1 Optimize server hardware
The hardware performance of the server directly determines the performance of the MySQL database. The performance bottleneck of the hardware directly determines the running speed and efficiency of the MySQL database. Improving hardware configuration for performance bottlenecks can increase the speed of MySQL database query and update. (1) Configure larger memory (2) Configure a high-speed disk system (3) Reasonably distribute disk I/O (4) Configure multiple processors
2.2 Optimize MySQL parameters
-
innodb_buffer_pool_size: This parameter is one of the most important parameters of the Mysql database, indicating the maximum cache of InnoDB type tables and indexes. It not only caches index data, but also caches table data. The larger this value is, the faster the query will be. But if this value is too large, it will affect the performance of the operating system. -
key_buffer_size: Indicates the size of the index buffer. The index buffer is shared by all threads. Increasing the index buffer results in better index handling (for all reads and multiple writes). Of course, the bigger the value, the better. Its size depends on the size of the memory. If this value is too large, it will cause the operating system to change pages frequently and reduce system performance. For servers with memory around 4GB, this parameter can be set to 256M or 384M. -
table_cache: Indicates the number of tables opened at the same time. The larger this value is, the more tables can be opened at the same time. The larger the physical memory, the larger the setting. The default is 2402, and it is best to adjust it to 512-1024. This value is not as large as possible, because too many tables opened at the same time will affect the performance of the operating system. -
query_cache_size: Indicates the size of the query buffer. It can be observed in the MySQL console. If the value of Qcache_lowmem_prunes is very large, it indicates that there is often insufficient buffering, so it is necessary to increase the value of Query_cache_size; if the value of Qcache_hits is very large, it indicates that the query buffer is used very frequently. If the value is larger than If the value is small, it will affect the efficiency, so you can consider not querying the cache; Qcache_free_blocks, if the value is very large, it indicates that there are many fragments in the buffer. Invalid after MySQL8.0. This parameter needs to be used in conjunction with query_cache_type -
query_cache_typeWhen the value is 0, all queries do not use the query cache. However, query_cache_type=0 will not cause MySQL to release the cache memory configured by query_cache_size.- When
query_cache_type=1, all queries will use the query cache unless SQL_NO_CACHE is specified in the query statement, such as SELECT SQL_NO_CACHE * FROM tbl_name - When
query_cache_type=2, the query will use the query cache only if the SQL_CACHE keyword is used in the query statement. Using the query cache can improve the speed of queries. This method is only suitable for situations where there are few modification operations and the same query operations are performed frequently.
- When
-
sort_buffer_size: Indicates the size of the buffer allocated by each thread that needs to be sorted. Increasing the value of this parameter can improve the speed of ORDER BY or GROUP BY operations. The default value is 2 097 144 bytes (approximately 2MB). For a server with a memory of about 4GB, the recommended setting is 6-8M. If there are 100 connections, the total allocated sort buffer size is 100 × 6 = 600MB. -
join_buffer_size= 8M: Indicates the buffer size that can be used for joint query operations. Like sort_buffer_size, the allocated memory corresponding to this parameter is also exclusive to each connection. -
read_buffer_size: Indicates the size (in bytes) of the buffer allocated for each table scanned by each thread when scanning continuously. This buffer is needed when the thread reads records continuously from the table. SET SESSION read_buffer_size=n can temporarily set the value of this parameter. The default is 64K and can be set to 4M -
innodb_flush_log_at_trx_commit: Indicates when the buffer data is written to the log file, and the log file is written to the disk. This parameter is very important for the innoDB engine. This parameter has 3 values, 0, 1 and 2. The default value of this parameter is 1 -
A value of 1 means that data is written to the log file every time a transaction is committed and the log file is written to disk for synchronization. This mode is the safest, but also the slowest. Because every transaction submission or instruction outside the transaction requires the log to be written (flush) to the hard disk.
-
When the value is 2, it means that data is written to the log file every time a transaction is committed, and the log file is written to the disk every 1 second. This mode is faster and safer than 0. Only when the operating system crashes or the system is powered off, all transaction data in the previous second may be lost.
-
innodb_log_buffer_size: This is the buffer used by the transaction log of the InnoDB storage engine. In order to improve performance, the information is also written to the Innodb Log Buffer first. When the corresponding conditions set by the innodb_flush_log_trx_commit parameter are met (or the log buffer is full), the log will be written to the file (or synchronized to the disk). -
max_connections: Indicates the maximum number of connections allowed to the MySQL database. The default value is 151. If the status variable connection_errors_max_connections is not zero and keeps growing, it means that connection requests continue to fail because the number of database connections has reached the maximum allowable value. In this case, you can consider increasing the value of max_connections. Under the Linux platform, it is not difficult for a server with good performance to support 500-1000 connections. It needs to be evaluated and set based on the server performance. The larger the number of connections, the better, because these connections will waste memory resources. Too many connections may cause the MySQL server to freeze -
back_log: Used to control the backlog request stack size set when MySQL listens to the TCP port. If the number of MySql connections reaches max_connections, new requests will be stored in the stack to wait for a certain connection to release resources. The number of the stack is back_log. If the number of waiting connections exceeds back_log, connection resources will not be granted. An error will be reported. The default value before version 5.6.6 is 50, and the default value for subsequent versions is 50 + (max_connections / 5). For Linux systems, it is recommended to set it to an integer less than 512, but the maximum does not exceed 900. If you need the database to handle a large number of connection requests in a short period of time, you can consider increasing the value of back_log appropriately. -
thread_cache_size: The size of the number of cached threads in the thread pool. When the client disconnects, the current thread is cached. When a new connection request is received, the response is fast without creating a new thread. This can greatly improve the efficiency of connection creation, especially for applications that use short connections. Then in order to improve performance, you can increase the value of this parameter. Default is 60, can be set to 120.
The size of the thread pool can be appropriately adjusted through the following MySQL status values:
mysql> show global status like 'Thread%';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Threads_cached | 2 |
| Threads_connected | 1 |
| Threads_created | 3 |
| Threads_running | 2 |
+-------------------+-------+
4 rows in set (0.01 sec)
When Threads_cached becomes less and less, but Threads_connected never decreases, and Threads_created continues to increase, the size of thread_cache_size can be increased appropriately.
- wait_timeout: Specify the maximum connection time for a request. For servers with about 4GB of memory, it can be set to 5-10
- interactive_timeout: Indicates the number of seconds the server waits for action before closing the connection
Here is a reference configuration of my.cnf:
[mysqld]
port = 3306 serverid = 1 socket = /tmp/mysql.sock skip-locking #避免MySQL的外部锁定，减少
出错几率增强稳定性。
skip-name-resolve #禁止MySQL对外部连接进行DNS解析，使用这一选项可以消除MySQL进行DNS解析的时间。但需要注意，如果开启该选项，则所有远程主机连接授权要使用IP地址方式，否则MySQL将无法正常处理连接请求！
back_log = 384
key_buffer_size = 256M
max_allowed_packet = 4M
thread_stack = 256K
table_cache = 128K
sort_buffer_size = 6M
read_buffer_size = 4M
read_rnd_buffer_size=16M
join_buffer_size = 8M
myisam_sort_buffer_size =64M
table_cache = 512
thread_cache_size = 64
query_cache_size = 64M
tmp_table_size = 256M
max_connections = 768
max_connect_errors = 10000000
wait_timeout = 10
thread_concurrency = 8 #该参数取值为服务器逻辑CPU数量*2，在本例中，服务器有2颗物理CPU，而每颗物理CPU又支持H.T超线程，所以实际取值为4*2=8
skipnetworking #开启该选项可以彻底关闭MySQL的TCP/IP连接方式，如果WEB服务器是以远程连接的方式访问MySQL数据库服务器则不要开启该选项！否则将无法正常连接！
table_cache=1024
innodb_additional_mem_pool_size=4M #默认为2M
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=2M #默认为1M
innodb_thread_concurrency=8 #你的服务器CPU有几个就设置为几。建议用默认一般为8
tmp_table_size=64M #默认为16M，调到64-256最挂
thread_cache_size=120
query_cache_size=32M
2.3 Case analysis
The e-commerce platform has too much data and the CPU usage is too high. The database needs to be tuned.
Adjust the system parameter InnoDB_flush_log_at_trx_commit
The default value is 1, change it to 2. There is no need to start disk reading and writing every time a transaction is submitted. In large concurrency scenarios, it can improve system efficiency and reduce CPU usage.
Adjust the system parameter InnoDB_buffer_pool_size
InnoDB storage engine is used 缓存来存储索引和数据. The larger the value, the more indexes and data that can be loaded into the cache area, and the less disk reading and writing is required. If the parameter is modified to 64G, the number of disk reading and writing can be greatly reduced, which can fully utilize the memory and release some CPU resources.
Adjust the system parameter InnoDB_buffer_pool_instances
Can improve the system's performance 并行处理能力by allowing multiple processes to process different parts of the cache at the same time
When encountering the problem of insufficient CPU resources, 2 ideas
- Clear congested road sections, eliminate bottlenecks, and shorten waiting times
- Open up new channels and increase parallel processing capabilities