mysql性能优化
1.1 MySQL服务的工作过程
管理工具：MySQL服务软件安装后提供的命令
连接池：检查本机是否有资源处理当前的连接请求 （空闲的线程 内存）
[root@55 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
sql接口：把sql命令传递给mysql服务的进程处理。
分析器：检查执行的sql命令是否有语法错误
优化器：优化执行的sql命令，使其能以最节省系统资源的方式执行。
查询缓存：查询缓存的存储空间是从系统的物理内存里划分出来的，用来存储查询过的查询结果。
存储引擎：软件自带的功能程序，是用来处理表的处理器。
文件系统：数据库服务器存储数据的磁盘
1.2 MySQL服务处理查询请求的过程、
MySQL性能优化
数据库服务器处理客户的连接慢，可能是由哪些原因导致。
1.监控服务器的监控信息
a.网络带宽
b.服务器硬件的配置：查看服务器硬件资源的使用情况 CPU 内存 存储 I/O
2.提供数据库服务软件版本低。
查看服务运行时的参数设置：
mysql> show variables;
帮助文档：
MySQL帮助手册
mysql配置文件详解
修改变量的值：
命令行修改
set [global] 变量名=值;
永久修改
vim /etc/my.cnf
变量名=值
常用的参数有哪些？
并发连接数控制
max_connections 容许的最大的并发
mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.00 sec)
mysql> set max_connections = 500;
ERROR 1229 (HY000): Variable 'max_connections' is a GLOBAL variable and should be set with SET GLOBAL
mysql> set global max_connections = 500;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 500 |
+-----------------+-------+
1 row in set (0.00 sec)
有过的最大连接量/并发连接数=0.85
留15%空闲是为了后期可以访问
show global status like "%count%";
mysql> show global status like "%conn%";
+-----------------------------------------------+---------------------+
| Variable_name | Value |
+-----------------------------------------------+---------------------+
| Aborted_connects | 0 |
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 4 |
| Locked_connects | 0 |
| Max_used_connections | 1 |
| Max_used_connections_time | 2018-06-30 09:53:28 |
| Performance_schema_session_connect_attrs_lost | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_finished_connects | 0 |
| Threads_connected | 1 |
+-----------------------------------------------+---------------------+
16 rows in set (0.00 sec)
Max_used_connections/Max_connections=0.85
mysql> flush status; ---清空连接的值
缓冲区、线程
可以重复使用的线程的数量
hread_cache_size =9
mysql> show variables like "%thread%"
-> ;
+-----------------------------------------+---------------------------+
| Variable_name | Value |
+-----------------------------------------+---------------------------+
| innodb_purge_threads | 4 |
| innodb_read_io_threads | 4 |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_write_io_threads | 4 |
| max_delayed_threads | 20 |
| max_insert_delayed_threads | 20 |
| myisam_repair_threads | 1 |
| performance_schema_max_thread_classes | 50 |
| performance_schema_max_thread_instances | -1 |
| pseudo_thread_id | 3 |
| thread_cache_size | 9 |
| thread_handling | one-thread-per-connection |
| thread_stack | 262144 |
+-----------------------------------------+---------------------------+
14 rows in set (0.00 sec)
mysql> show variables like "%table%";
+----------------------------------------+----------+
| Variable_name | Value |
+----------------------------------------+----------+
| big_tables | OFF |
| innodb_file_per_table | ON |
| innodb_ft_aux_table | |
| innodb_ft_server_stopword_table | |
| innodb_ft_user_stopword_table | |
| innodb_table_locks | ON |
| innodb_undo_tablespaces | 0 |
| lower_case_table_names | 0 |
| max_heap_table_size | 16777216 |
| max_tmp_tables | 32 |
| old_alter_table | OFF |
| performance_schema_max_table_handles | -1 |
| performance_schema_max_table_instances | -1 |
| performance_schema_max_table_lock_stat | -1 |
| table_definition_cache | 1400 |
| table_open_cache | 2000 |
| table_open_cache_instances | 16 |
| tmp_table_size | 16777216 |
| updatable_views_with_limit | YES |
+----------------------------------------+----------+
19 rows in set (0.01 sec)
为所有线程缓存打开表的数量
table_open_cache | 2000
key_buffer_size 索引缓存大小
index primary key foreign key
字段值排序 存储在 a.MYI /var/lib/mysql/
a.ibd
key_buffer_size | 8388608
sort_buffer_size 为每个排序的线程分配此大小的缓存空间
read_buffer_size 为顺序读取表记录保留的缓存大小
查看数据库缓存的设置
mysql> show variables like "%cache%";
+--------------------------------+----------------------+
| Variable_name | Value |
+--------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_stmt_cache_size | 32768 |
| have_query_cache | YES |
| host_cache_size | 279 |
| innodb_disable_sort_file_cache | OFF |
| innodb_ft_cache_size | 8000000 |
| innodb_ft_result_cache_limit | 2000000000 |
| innodb_ft_total_cache_size | 640000000 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| metadata_locks_cache_size | 1024 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 1048576 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
| stored_program_cache | 256 |
| table_definition_cache | 1400 |
| table_open_cache | 2000 |
| table_open_cache_instances | 16 |
| thread_cache_size | 9 |
+--------------------------------+----------------------+
24 rows in set (0.01 sec)
query_cache_size | 1048576
实际的环境中是不开查询缓存的
mysql> show variables like "%query_cache%";
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| have_query_cache | YES |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 1048576 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+
6 rows in set (0.01 sec)
query_cache_type =0|1|2
0不开
1 把查询的结果放到查询缓存中，但是不能超过放入缓存的限制query_cache_limit
2 如果需要把查询的结果放到查询都缓存中，必须加select sql_in_cache 字段名列表from表；
query_cache_size 缓存大小
query_cache_min_res_unit 最小存储单元
query_cache_wlock_invalidate 查询缓存结果无效关闭
当对myisam存储引擎的表，查询的时候，若此时有客户端对表执行写操作，mysql服务不会从缓存里查找数据返回给客户端，而是等写操作完成后，重新从表里查找数据返回给客户端。
脏读
mysql> show global status like "qcache%"
-> ;
+-------------------------+---------+
| Variable_name | Value |
+-------------------------+---------+
| Qcache_free_blocks | 1 |
| Qcache_free_memory | 1031832 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 0 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 1 |
+-------------------------+---------+
8 rows in set (0.00 sec)
查询缓存统计信息
| Qcache_hits | 0 | ----查询缓存的数量
| Qcache_inserts | 0 | ----查询结果总数
Qcache_lowmem_prunes ----- 最低的查询内存
Qcache_not_cached -----不让加入查询结果的次数
3.程序员编写的访问数据库服务数据的sql命令复杂，导致处理的速度缓慢。
在数据库服务器上启动慢查询日志，记录超过指定时间显示查询结果的sql命令。
binlog日志 错误日志 查询日志 慢查询日志
vim /etc/my.cnf
log-error=/var/log/mysqld.log
下午课程
查询日志：记录所有的sql命令
启用日日志 general-log
日志名 主机名.log
[root@55 ~]# vim /etc/my.cnf
[mysqld]
general-log
[root@55 ~]# systemctl restart mysqld
[root@55 ~]# cd /var/lib/mysql
[root@55 mysql]# ls
55.log client-cert.pem ib_logfile0 master51.index performance_schema server-key.pem
auto.cnf client-key.pem ib_logfile1 mysql private_key.pem sys
ca-key.pem ib_buffer_pool ibtmp1 mysql.sock public_key.pem
ca.pem ibdata1 master51.000001 mysql.sock.lock server-cert.pem
[root@55 mysql]# cat 55.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
2018-06-30T06:34:26.898335Z 1 Query SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE CREATE_OPTIONS LIKE '%partitioned%';
慢查询日志：只记录超过超时时间显示查询结果的sql命令
[root@55 mysql]# vim /etc/my.cnf
slow-query-log
log_queries_not_using_indexes=1
[root@55 mysql]# systemctl restart mysqld
[root@55 mysql]# pwd
/var/lib/mysql
[root@55 mysql]# ls *.log
55.log 55-slow.log
[root@55 mysql]# cat 55-slow.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
mysql> select sleep(11);
+-----------+
| sleep(11) |
+-----------+
| 0 |
+-----------+
1 row in set (11.00 sec)
mysql> exit
Bye
[root@55 mysql]# cat 55-slow.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
# Time: 2018-06-30T06:57:50.894775Z
# User@Host: root[root] @ localhost [] Id: 3
# Query_time: 11.000309 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1530341870;
select sleep(11);
显示慢查询记录的命令
[root@55 mysql]# mysqldumpslow 55-slow.log
Reading mysql slow query log from 55-slow.log
Count: 1 Time=0.00s (0s) Lock=0.00s (0s) Rows=0.0 (0), 0users@0hosts
# Time: N-N-30T06:N:N.894775Z
# User@Host: root[root] @ localhost [] Id: N
# Query_time: N.N Lock_time: N.N Rows_sent: N Rows_examined: N
SET timestamp=N;
select sleep(N)
SQL查询优化
软优化 --改变参数
升级硬件
MySQL数据读写分离
MySQL多实例
1.1 MySQL服务的工作过程
管理工具：MySQL服务软件安装后提供的命令
连接池：检查本机是否有资源处理当前的连接请求 （空闲的线程 内存）
[root@55 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.17 MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
sql接口：把sql命令传递给mysql服务的进程处理。
分析器：检查执行的sql命令是否有语法错误
优化器：优化执行的sql命令，使其能以最节省系统资源的方式执行。
查询缓存：查询缓存的存储空间是从系统的物理内存里划分出来的，用来存储查询过的查询结果。
存储引擎：软件自带的功能程序，是用来处理表的处理器。
文件系统：数据库服务器存储数据的磁盘
1.2 MySQL服务处理查询请求的过程、
MySQL性能优化
数据库服务器处理客户的连接慢，可能是由哪些原因导致。
1.监控服务器的监控信息
a.网络带宽
b.服务器硬件的配置：查看服务器硬件资源的使用情况 CPU 内存 存储 I/O
2.提供数据库服务软件版本低。
查看服务运行时的参数设置：
mysql> show variables;
帮助文档：
MySQL帮助手册
mysql配置文件详解
修改变量的值：
命令行修改
set [global] 变量名=值;
永久修改
vim /etc/my.cnf
变量名=值
常用的参数有哪些？
并发连接数控制
max_connections 容许的最大的并发
mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.00 sec)
mysql> set max_connections = 500;
ERROR 1229 (HY000): Variable 'max_connections' is a GLOBAL variable and should be set with SET GLOBAL
mysql> set global max_connections = 500;
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like "max_connections";
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 500 |
+-----------------+-------+
1 row in set (0.00 sec)
有过的最大连接量/并发连接数=0.85
留15%空闲是为了后期可以访问
show global status like "%count%";
mysql> show global status like "%conn%";
+-----------------------------------------------+---------------------+
| Variable_name | Value |
+-----------------------------------------------+---------------------+
| Aborted_connects | 0 |
| Connection_errors_accept | 0 |
| Connection_errors_internal | 0 |
| Connection_errors_max_connections | 0 |
| Connection_errors_peer_address | 0 |
| Connection_errors_select | 0 |
| Connection_errors_tcpwrap | 0 |
| Connections | 4 |
| Locked_connects | 0 |
| Max_used_connections | 1 |
| Max_used_connections_time | 2018-06-30 09:53:28 |
| Performance_schema_session_connect_attrs_lost | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_finished_connects | 0 |
| Threads_connected | 1 |
+-----------------------------------------------+---------------------+
16 rows in set (0.00 sec)
Max_used_connections/Max_connections=0.85
mysql> flush status; ---清空连接的值
缓冲区、线程
可以重复使用的线程的数量
hread_cache_size =9
mysql> show variables like "%thread%"
-> ;
+-----------------------------------------+---------------------------+
| Variable_name | Value |
+-----------------------------------------+---------------------------+
| innodb_purge_threads | 4 |
| innodb_read_io_threads | 4 |
| innodb_thread_concurrency | 0 |
| innodb_thread_sleep_delay | 10000 |
| innodb_write_io_threads | 4 |
| max_delayed_threads | 20 |
| max_insert_delayed_threads | 20 |
| myisam_repair_threads | 1 |
| performance_schema_max_thread_classes | 50 |
| performance_schema_max_thread_instances | -1 |
| pseudo_thread_id | 3 |
| thread_cache_size | 9 |
| thread_handling | one-thread-per-connection |
| thread_stack | 262144 |
+-----------------------------------------+---------------------------+
14 rows in set (0.00 sec)
mysql> show variables like "%table%";
+----------------------------------------+----------+
| Variable_name | Value |
+----------------------------------------+----------+
| big_tables | OFF |
| innodb_file_per_table | ON |
| innodb_ft_aux_table | |
| innodb_ft_server_stopword_table | |
| innodb_ft_user_stopword_table | |
| innodb_table_locks | ON |
| innodb_undo_tablespaces | 0 |
| lower_case_table_names | 0 |
| max_heap_table_size | 16777216 |
| max_tmp_tables | 32 |
| old_alter_table | OFF |
| performance_schema_max_table_handles | -1 |
| performance_schema_max_table_instances | -1 |
| performance_schema_max_table_lock_stat | -1 |
| table_definition_cache | 1400 |
| table_open_cache | 2000 |
| table_open_cache_instances | 16 |
| tmp_table_size | 16777216 |
| updatable_views_with_limit | YES |
+----------------------------------------+----------+
19 rows in set (0.01 sec)
为所有线程缓存打开表的数量
table_open_cache | 2000
key_buffer_size 索引缓存大小
index primary key foreign key
字段值排序 存储在 a.MYI /var/lib/mysql/
a.ibd
key_buffer_size | 8388608
sort_buffer_size 为每个排序的线程分配此大小的缓存空间
read_buffer_size 为顺序读取表记录保留的缓存大小
查看数据库缓存的设置
mysql> show variables like "%cache%";
+--------------------------------+----------------------+
| Variable_name | Value |
+--------------------------------+----------------------+
| binlog_cache_size | 32768 |
| binlog_stmt_cache_size | 32768 |
| have_query_cache | YES |
| host_cache_size | 279 |
| innodb_disable_sort_file_cache | OFF |
| innodb_ft_cache_size | 8000000 |
| innodb_ft_result_cache_limit | 2000000000 |
| innodb_ft_total_cache_size | 640000000 |
| key_cache_age_threshold | 300 |
| key_cache_block_size | 1024 |
| key_cache_division_limit | 100 |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| metadata_locks_cache_size | 1024 |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 1048576 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
| stored_program_cache | 256 |
| table_definition_cache | 1400 |
| table_open_cache | 2000 |
| table_open_cache_instances | 16 |
| thread_cache_size | 9 |
+--------------------------------+----------------------+
24 rows in set (0.01 sec)
query_cache_size | 1048576
实际的环境中是不开查询缓存的
mysql> show variables like "%query_cache%";
+------------------------------+---------+
| Variable_name | Value |
+------------------------------+---------+
| have_query_cache | YES |
| query_cache_limit | 1048576 |
| query_cache_min_res_unit | 4096 |
| query_cache_size | 1048576 |
| query_cache_type | OFF |
| query_cache_wlock_invalidate | OFF |
+------------------------------+---------+
6 rows in set (0.01 sec)
query_cache_type =0|1|2
0不开
1 把查询的结果放到查询缓存中，但是不能超过放入缓存的限制query_cache_limit
2 如果需要把查询的结果放到查询都缓存中，必须加select sql_in_cache 字段名列表from表；
query_cache_size 缓存大小
query_cache_min_res_unit 最小存储单元
query_cache_wlock_invalidate 查询缓存结果无效关闭
当对myisam存储引擎的表，查询的时候，若此时有客户端对表执行写操作，mysql服务不会从缓存里查找数据返回给客户端，而是等写操作完成后，重新从表里查找数据返回给客户端。
脏读
mysql> show global status like "qcache%"
-> ;
+-------------------------+---------+
| Variable_name | Value |
+-------------------------+---------+
| Qcache_free_blocks | 1 |
| Qcache_free_memory | 1031832 |
| Qcache_hits | 0 |
| Qcache_inserts | 0 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 0 |
| Qcache_queries_in_cache | 0 |
| Qcache_total_blocks | 1 |
+-------------------------+---------+
8 rows in set (0.00 sec)
查询缓存统计信息
| Qcache_hits | 0 | ----查询缓存的数量
| Qcache_inserts | 0 | ----查询结果总数
Qcache_lowmem_prunes ----- 最低的查询内存
Qcache_not_cached -----不让加入查询结果的次数
3.程序员编写的访问数据库服务数据的sql命令复杂，导致处理的速度缓慢。
在数据库服务器上启动慢查询日志，记录超过指定时间显示查询结果的sql命令。
binlog日志 错误日志 查询日志 慢查询日志
vim /etc/my.cnf
log-error=/var/log/mysqld.log
下午课程
查询日志：记录所有的sql命令
启用日日志 general-log
日志名 主机名.log
[root@55 ~]# vim /etc/my.cnf
[mysqld]
general-log
[root@55 ~]# systemctl restart mysqld
[root@55 ~]# cd /var/lib/mysql
[root@55 mysql]# ls
55.log client-cert.pem ib_logfile0 master51.index performance_schema server-key.pem
auto.cnf client-key.pem ib_logfile1 mysql private_key.pem sys
ca-key.pem ib_buffer_pool ibtmp1 mysql.sock public_key.pem
ca.pem ibdata1 master51.000001 mysql.sock.lock server-cert.pem
[root@55 mysql]# cat 55.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
2018-06-30T06:34:26.898335Z 1 Query SELECT TABLE_SCHEMA, TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE CREATE_OPTIONS LIKE '%partitioned%';
慢查询日志：只记录超过超时时间显示查询结果的sql命令
[root@55 mysql]# vim /etc/my.cnf
slow-query-log
log_queries_not_using_indexes=1
[root@55 mysql]# systemctl restart mysqld
[root@55 mysql]# pwd
/var/lib/mysql
[root@55 mysql]# ls *.log
55.log 55-slow.log
[root@55 mysql]# cat 55-slow.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
mysql> select sleep(11);
+-----------+
| sleep(11) |
+-----------+
| 0 |
+-----------+
1 row in set (11.00 sec)
mysql> exit
Bye
[root@55 mysql]# cat 55-slow.log
/usr/sbin/mysqld, Version: 5.7.17-log (MySQL Community Server (GPL)). started with:
Tcp port: 0 Unix socket: /var/lib/mysql/mysql.sock
Time Id Command Argument
# Time: 2018-06-30T06:57:50.894775Z
# User@Host: root[root] @ localhost [] Id: 3
# Query_time: 11.000309 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 0
SET timestamp=1530341870;
select sleep(11);
显示慢查询记录的命令
[root@55 mysql]# mysqldumpslow 55-slow.log
Reading mysql slow query log from 55-slow.log
Count: 1 Time=0.00s (0s) Lock=0.00s (0s) Rows=0.0 (0), 0users@0hosts
# Time: N-N-30T06:N:N.894775Z
# User@Host: root[root] @ localhost [] Id: N
# Query_time: N.N Lock_time: N.N Rows_sent: N Rows_examined: N
SET timestamp=N;
select sleep(N)
SQL查询优化
软优化 --改变参数
升级硬件
MySQL数据读写分离
MySQL多实例