Article Directory
MySQL bottleneck optimization
(Transformation of business from small to large) Assuming that a website has a small number of visitors from the very beginning to reach tens of millions of PV per day, let's speculate on the evolution of its mysql server architecture.
The first stage: The daily pv level of website visits is below 1w. A single machine runs web and db, and there is no need to do architecture layer tuning (for example, there is no need to increase the memcached cache). At this time, the data is often cold backed up daily, but sometimes if data security is considered, a mysql master-slave will be built.
The second stage: The daily pv of website visits reaches tens of thousands. At this time, a single machine is already a bit loaded, we need to separate web and db, and we need to build a memcached service as a cache. In other words, at this stage, we can also use a single machine to run mysql to undertake the data storage and query of the entire website. The purpose of doing mysql master-slave is also for data security.
The third stage: The daily pv of website visits reaches hundreds of thousands. Although a single machine can also be supported, the machine configuration required is much better than the previous machine. If the funds allow, you can buy a machine with a high configuration to run the mysql service, but it does not mean that the configuration is doubled. At a certain stage, the increase in the configuration can no longer bring about the increase in performance. Therefore, at this stage, we will think of doing mysql service clusters, which means we can run mysql with multiple machines. But the mysql cluster is different from the web cluster. We need to consider the consistency of the data, so we can't simply apply the way of doing the web cluster. The architecture that can be done is mysql master-slave, one master and multiple slaves. In order to ensure the robustness of the architecture and the integrity of the data, there can only be one master and multiple slaves. There is another problem. We need to think about it. In the front-end web layer, we specify the ip of the mysql machine in our program. Then, when there are multiple mysql machines, how to configure it in the program? We can use multiple machines to run mysql, one of them is for writing, and the others are for reading. We only need to configure the read and write ip into the program, and the program will automatically distinguish the machines.
The fourth stage: The daily pv of website visits reaches several million. The previous one-master-multi-slave model has encountered a bottleneck, because when the website visits increase, the amount of reading the database will also increase. We need to add more slaves, but when the number of slaves increases to dozens, the master It is necessary to transfer all the bin-logs from above, so this process itself is a very cumbersome reading, which will inevitably cause a great delay in the data synchronized from above. Therefore, we can make an optimization. Make MySQL's original one master and multiple slaves become one master and one slave, and then serve as the master of other slaves. The former master is only responsible for the writing of website business, while the latter is never responsible for any website. The business is only responsible for synchronizing bin-log to other slaves. In this way, you can continue to stack several more slave libraries.
Phase 5: When the number of daily pv on the website reached 10 million, we found that the amount of writing on the website was very large. There was only one master in our previous architecture, and the master here has become a bottleneck. Therefore, further adjustments are needed.
Optimize information query
Solve the bottleneck: use show status to query performance parameters of some MySQL databases for indexing
//Number of times to connect to the mysql server
mysql> show status like "connections";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Connections | 3 |
+---------------+-------+
1 row in set (0.02 sec)
//The online time of the mysql server
mysql> show status like "uptime";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Uptime | 177 |
+---------------+-------+
1 row in set (0.00 sec)
//Number of query operations
mysql> show status like "com_select";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Com_select | 1 |
+---------------+-------+
1 row in set (0.00 sec)
//Number of update operations
mysql> show status like "com_update";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Com_update | 0 |
+---------------+-------+
1 row in set (0.00 sec)
//Number of insert operations
mysql> show status like "com_insert";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Com_insert | 0 |
+---------------+-------+
1 row in set (0.00 sec)
//Number of delete operations
mysql> show status like "com_delete";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Com_delete | 0 |
+---------------+-------+
1 row in set (0.00 sec)
//Number of slow queries to query the mysql server
mysql> show status like "slow_queries";
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Slow_queries | 0 |
+---------------+-------+
1 row in set (0.00 sec)
Analyze query statements
语法：Explain [extended] select select_option;
mysql> explain select * from fruits\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL //使用哪个列或常数与索引一起使用来查询记录
rows: 16
filtered: 100.00
Extra: NULL
1 row in set, 1 warning (0.01 sec)
Select_type: indicates the type of select statement, where simple is a simple query (not including join queries and subqueries) Primary main query Union join query
Type: The connection type of the table System only has one row in the system table. Const data table has at most one row that matches, which will be read at the beginning of the query and used as a constant Eq_ref for comparison with the index in the rest of the query optimization. Column ref For any combination of rows from the previous table, read all matching rows from the table ref_or_null Same as above, add a row index_merge that can specifically search for a null value. The connection type indicates that the index is used and the optimization method range only retrieves The row index of a given range has the same connection type as all, except that only any combination of rows in the table before the index tree all is scanned, and a complete table scan is performed
possible_keys: NULL //Point out which index mysql uses to find a row in the table. NULL means that no index is created
//Or use desc
Same function as explain
mysql> desc select * from fruits\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 100.00
Extra: NULL
1 row in set, 1 warning (0.00 sec)
//Index improves query efficiency Create index index name on table name (field)
mysql> explain select * from fruits where f_name='apple'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 10.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
mysql> create index index_t1 on fruits(f_name);
Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> explain select * from fruits where f_name='apple'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ref
possible_keys: index_t1
key: index_t1
key_len: 765
ref: const
rows: 1
filtered: 100.00
Extra: NULL
1 row in set, 1 warning (0.00 sec)
//Optimize query speed
//Note: 1 After indexing, use like'xx%'% not in the first place to query the most efficient
mysql> explain select * from fruits where f_name='apple' like '%x'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: index_t1
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 10.00
Extra: Using where
1 row in set, 3 warnings (0.00 sec)
mysql> explain select * from fruits where f_name='apple' like '%x'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: index_t1
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 10.00
Extra: Using where
1 row in set, 3 warnings (0.00 sec)
//表里数据太少不太好验证上下都是(0.00 sec)
//2 Multi-field index, except for the first field to query the fastest, the rest do not follow the index, the index does not take effect
mysql> create index index_t2 on fruits(f_id,f_price);
Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> explain select * from fruits where f_id='a1'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: const
possible_keys: PRIMARY,index_t2
key: PRIMARY
key_len: 30
ref: const
rows: 1
filtered: 100.00
Extra: NULL
1 row in set, 1 warning (0.00 sec)
mysql> explain select * from fruits where f_price='3.60'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 10.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
//3 If the field set by the index is created, the left and right values of the query index combination or are all the values under the index setting field
mysql> explain select * from fruits where f_name='apple' or s_id=101\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: ALL
possible_keys: index_t1
key: NULL
key_len: NULL
ref: NULL
rows: 16
filtered: 19.00
Extra: Using where
1 row in set, 1 warning (0.00 sec)
mysql> explain select * from fruits where f_name='apple' or f_id='12'\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: fruits
partitions: NULL
type: index_merge
possible_keys: PRIMARY,index_t1,index_t2
key: index_t1,PRIMARY
key_len: 765,30
ref: NULL
rows: 2
filtered: 100.00
Extra: Using union(index_t1,PRIMARY); Using where
1 row in set, 1 warning (0.00 sec)
profiling analysis query
Through slow log query, you can know which SQL statements are inefficiently executed. Through explain, we can learn the specific execution of SQL statements, index usage, etc., and you can also view the execution status in conjunction with the show command. If you feel that the explain information is not detailed enough, you can use the profiling command to get more accurate information about the system resources consumed by SQL execution. Profiling is turned off by default. Can be viewed by the following statement
mysql> show variables like 'profiling';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| profiling | OFF | //off表示未开启
+---------------+-------+
1 row in set (0.00 sec)
mysql> select @@profiling;
+-------------+
| @@profiling |
+-------------+
| 0 | //0表示未开启
+-------------+
1 row in set, 1 warning (0.00 sec)
//Open the profiling function
mysql> set profiling=1;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> select @@profiling;
+-------------+
| @@profiling |
+-------------+
| 1 |
+-------------+
1 row in set, 1 warning (0.00 sec)
//Execute the sql statement to be tested
mysql> select * from fruits where f_name='apple';
+------+------+--------+---------+
| f_id | s_id | f_name | f_price |
+------+------+--------+---------+
| a1 | 101 | apple | 5.20 |
+------+------+--------+---------+
1 row in set (0.00 sec)
mysql> show profiles;
+----------+------------+--------------------------------------------------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+--------------------------------------------------------------------------------+
| 32 | 0.00126600 | show status like 'slow_query' |
| 33 | 0.00092000 | show status like 'slow_query%' |
| 34 | 0.00098425 | show status like '%slow_query%' |
| 35 | 0.00143000 | show variables like '%slow_query%' |
| 36 | 0.00145650 | show variables like '%slow_quer%' |
| 37 | 0.00097325 | show status like '%slow_quer%' |
| 38 | 0.00022850 | help 'tirgger' |
| 39 | 0.01394825 | help 'create' |
| 40 | 0.00044375 | help 'CREATE TRIGGER' |
| 41 | 0.02160925 | create table goods
(
g_id int,
name varchar(22),
quantity int
) |
| 42 | 0.00876225 | create table orders
(
o_id int,
g_id int,
counts int
) |
| 43 | 0.00137850 | show variables like "%profiling%" |
| 44 | 0.00011625 | select @@profiling |
| 45 | 0.00012150 | select @@profiling |
| 46 | 0.00032050 | select * from fruits where f_name='apple' |
+----------+------------+--------------------------------------------------------------------------------+
15 rows in set, 1 warning (0.00 sec)
mysql> show profile for query 46; //查看46条语句详细信息
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000060 |
| checking permissions | 0.000006 |
| Opening tables | 0.000016 |
| init | 0.000029 |
| System lock | 0.000010 |
| optimizing | 0.000045 |
| statistics | 0.000065 |
| preparing | 0.000011 |
| executing | 0.000003 |
| Sending data | 0.000039 |
| end | 0.000004 |
| query end | 0.000006 |
| closing tables | 0.000005 |
| freeing items | 0.000012 |
| cleaning up | 0.000012 |
+----------------------+----------+
15 rows in set, 1 warning (0.00 sec)
status: is the status in the profile
duration: is the time consuming in the status state. So what we focus on is which state is the most time-consuming, and which of these states can be optimized. Of course, you can also view more information such as CPU, etc.
语法：SHOW PROFILE [type [, type] … ][FOR QUERY n] type:
ALL: Display all overhead information
BLOCK IO: Display block IO related overhead
CPU: Display user CPU time, system CPU time
IPC: Display transmission and reception related overhead information
PAGE FAULTS: Display overhead information related to page faults
SWAPS: Display information about the overhead related to the number of exchanges
//After the test is completed, remember to turn off the debugging function, so as not to affect the normal use of the database:
mysql> set profiling=0;
Optimize the database table structure. Decompose a table with many fields into multiple tables. View the table structure of the table
Optimize the table structure 1 Decompose a table with a lot of fields into multiple tables 2 Add an intermediate table 3 Reasonably increase redundant fields 4 Optimize the speed of inserting records.
(1) Disable the index (disable the index before inserting the data, the creation of the index will not take effect, the command: alter table table name disable keys, note that the engine=myisam is added after the table is created, it can be disabled successfully)
mysql> alter table fruits disable keys; #禁用索引
Query OK, 0 rows affected, 1 warning (0.00 sec)
(2) Disable the uniqueness check (disable the uniqueness check before inserting the record, command: set unique_checks=0 close set unique_checks=1)
mysql> set unique_checks=0;
Query OK, 0 rows affected (0.00 sec)
(3) Use batch insert (multiple insert commands are integrated into one command)
mysql> INSERT INTO fruits values('x2','101','mongo2','5.5');
Query OK, 1 row affected (0.00 sec)
mysql>
mysql> INSERT INTO fruits values('x3','101','mongo2','5.5');
Query OK, 1 row affected (0.00 sec)
mysql>
mysql> INSERT INTO fruits values('x4','101','mongo2','5.5');
Query OK, 1 row affected (0.00 sec)
mysql>
mysql> insert into fruits values ('x8','101','mongo2','5.5'),
-> ('x7','101','mongo2','5.5'), ('x6','101','mongo2','5.5'),
-> ('x5','101','mongo2','5.5');
Query OK, 4 rows affected (0.00 sec)
Records: 4 Duplicates: 0 Warnings: 0
(4) Use load data infile batch insert for innnodb table (1) Disable uniqueness check
mysql> set unique_checks=0;
Query OK, 0 rows affected (0.00 sec)
(2) Disable foreign key checking (command: set foreign_key_checks=0, enable=1)
mysql> SET foreign_key_checks=0;
Query OK, 0 rows affected (0.00 sec)
(3) Disable autocommit (command: set autocommit=0, enable=1)
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
Analysis table, check table, optimization table
Analysis tables, checklists and optimization Table Analysis Table : Analysis of key distribution checklist : Check for errors optimized table : eliminate wasted space resulting from deleting or updating
Analyze the table statement : analyze [local |no_wirte_to_binlog] table tb1_name[tb2_name]...... The local keyword is not written into the binary log followed by one table or multiple tables. During the analysis period, it can only be read and cannot be inserted and updated.
mysql> analyze table fruits;
+---------------+---------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------------+---------+----------+----------+
| mytest.fruits | analyze | status | OK |
+---------------+---------+----------+----------+
1 row in set (0.00 sec)
Table is the operation performed by the table name op. msg_type Information level (status is normal status, info is information, note, warning, error error) msg_text is display information
Check table : check for errors, keyword statistics, check the view for errors. Check table table name option = {quick |fast | medium|extended |changed} Quick does not scan rows, does not check the wrong connection Fast only checks that it has not been closed properly The table Medium scans rows to verify that the deleted connection is valid, and you can also calculate the key checksum of each row. Extended performs a comprehensive keyword search for all keywords in each row. Changed only checks the tables that have been changed since the last check and the tables that have not been closed properly. Option is only valid for myisam and invalid for innodb tables. Read-only is added to the table during execution lock
mysql> check table fruits;
+---------------+-------+----------+----------+
| Table | Op | Msg_type | Msg_text |
+---------------+-------+----------+----------+
| mytest.fruits | check | status | OK |
+---------------+-------+----------+----------+
1 row in set (0.01 sec)
**Optimize the table: ** Eliminate the waste of space caused by delete or update Optimize [local |no_write_to_binlog] table tb1_name …. Only the myisam table and the innodb table can be optimized, but only the varchar\text\blob in the table can be optimized. Read-only lock during the process
mysql> optimize table fruits\G
*************************** 1. row ***************************
Table: mytest.fruits
Op: optimize
Msg_type: note
Msg_text: Table does not support optimize, doing recreate + analyze instead
*************************** 2. row ***************************
Table: mytest.fruits
Op: optimize
Msg_type: status
Msg_text: OK
2 rows in set (0.05 sec)