MySQL optimization practice hundreds of millions of large tables turn

Author: Jiaxin

Source: http://www.cnblogs.com/YangJiaXin/

background
analysis test
- select xxx_record statement
- delete xxx_record statement
Implement
After the Index Tuning
delete large table is optimized for small batch delete
to sum up

background

Example XX (a master-slave) in the morning every day XXX alarm message SLA alarm which means that there is a certain delay from the master (if it occurs at this time from the switching master, takes a long time before they can complete the handover, the delay to recover to ensure consistency of data from the master)

Up slow query XX number of instances (the execution time of more than 1s sql will be recorded), XX application that party every night doing the task deletion of data a month ago

analysis

Use pt-query-digest analysis tools in the latest week-slow.log MySQL
pt-query-digest --since = MySQL-slow.log 148H | less
first partial results

Slow query execution time recently spent a week, recorded a total of 25403s, slow sql maximum execution time of 266s, the average number of lines per slow sql execution time 5s, an average of 17.66 million scans

Results of the second part

The number of slow queries select arrival_record operation records are up to 40,000 times, with an average response time of 4s, delete arrival_record record six times, with an average response time of 258s

select xxx_record statement

select arrival_record slow query is similar to the following, the parameter field where clause is the same, different values of parameter passed
select count (*) from arrival_record where product_id = 26 and receive_time between '2019-03-25 14 : 00: 00 'and' 2019-03-25 15:00:00 'and receive_spend_ms> = 0 \ G

select arrival_record statement the number of rows in mysql Scan up to 56 million, the average number of lines scanned 172 million and concluded that due to the long execution times more than the number of lines scanned due

View the execution plan

explain select count(*) from arrival_record where product_id=26 and receive_time between '2019-03-25 14:00:00' and '2019-03-25 15:00:00' and receive_spend_ms>=0\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: arrival_record
partitions: NULL
type: ref
possible_keys: IXFK_arrival_record
key: IXFK_arrival_record
key_len: 8
ref: const
rows: 32261320
filtered: 3.70
Extra: Using index condition; Using where
1 row in set, 1 warning (0.00 sec)

Use the index IXFK_arrival_record, but the number of lines is expected to have a lot of scanning lines more than 3000 w

show index from arrival_record;
+----------------+------------+---------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+----------------+------------+---------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| arrival_record | 0 | PRIMARY | 1 | id | A | 107990720 | NULL | NULL | | BTREE | | |
| arrival_record | 1 | IXFK_arrival_record | 1 | product_id | A | 1344 | NULL | NULL | | BTREE | | |
| arrival_record | 1 | IXFK_arrival_record | 2 | station_no | A | 22161 | NULL | NULL | YES | BTREE | | |
| arrival_record | 1 | IXFK_arrival_record | 3 | sequence | A | 77233384 | NULL | NULL | | BTREE | | |
| arrival_record | 1 | IXFK_arrival_record | 4 | receive_time | A | 65854652 | NULL | NULL | YES | BTREE | | |
| arrival_record | 1 | IXFK_arrival_record | 5 | arrival_time | A | 73861904 | NULL | NULL | YES | BTREE | | |
+----------------+------------+---------------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

show create table arrival_record;
..........
arrival_spend_ms bigint(20) DEFAULT NULL,
total_spend_ms bigint(20) DEFAULT NULL,
PRIMARY KEY (id),
KEY IXFK_arrival_record (product_id,station_no,sequence,receive_time,arrival_time) USING BTREE,
CONSTRAINT FK_arrival_record_product FOREIGN KEY (product_id) REFERENCES product (id) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=614538979 DEFAULT CHARSET=utf8 COLLATE=utf8_bin |

The table records the total number of approximately more than 100 million, the table has only one composite index, the product_id field small base, good selectivity

Incoming filter condition where product_id = 26 and receive_time between ' 2019-03-25 14:00:00' and '2019-03-25 15:00:00' and receive_spend_ms> = 0 no station_nu field, use less complex the index IXFK_arrival_record product_id, station_no, sequence, receive_time these fields

The leftmost prefix principle, select arrival_record uses only the first field product_id IXFK_arrival_record the composite index, and the field poor selectivity, resulting in many of the number of scanning lines, execution time

Receive_time field large cardinality, selectivity, this field may be indexed separately, select arrival_record sql will use the index

Now that you know select arrival_record where sentence passed in parameter field recorded in the slow query has product_id, receive_time, receive_spend_ms, you want to know there is no access to the table to filter through to other fields?

Artifact tcpdump out of time to

Use tcpdump packet capture for some time select statement to the table

tcpdump -i bond0 -s 0 -l -w - dst port 3316 | strings | grep select | egrep -i 'arrival_record' >/tmp/select_arri.log

Where to obtain a conditional statement later in the select statement from

IFS_OLD=$IFS
IFS=$'\n'
for i in `cat /tmp/select_arri.log `;do echo ${i#*'from'}; done | less IFS=$IFS_OLD

arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=17 and arrivalrec0_.station_no='56742'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=22 and arrivalrec0_.station_no='S7100'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=24 and arrivalrec0_.station_no='V4631'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=22 and arrivalrec0_.station_no='S9466'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=24 and arrivalrec0_.station_no='V4205'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=24 and arrivalrec0_.station_no='V4105'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=24 and arrivalrec0_.station_no='V4506'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=24 and arrivalrec0_.station_no='V4617'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=22 and arrivalrec0_.station_no='S8356'
arrival_record arrivalrec0_ where arrivalrec0_.sequence='2019-03-27 08:40' and arrivalrec0_.product_id=22 and arrivalrec0_.station_no='S8356'

The table select conditions where there product_id, station_no, sequence fields, the first three fields can be used to the composite index IXFK_arrival_record

Fully shown, the optimization method, remove the composite index IXFK_arrival_record, establish a composite index idx_sequence_station_no_product_id, and create separate index indx_receive_time

delete xxx_record statement

The average number of scanning lines of the delete operation is 1.1 million lines, the average execution time is 262s

delete statement is as follows, slow query log every incoming parameter value is not the same

delete from arrival_record where receive_time < STR_TO_DATE('2019-02-23', '%Y-%m-%d')\G

Implementation plan

explain select * from arrival_record where receive_time < STR_TO_DATE('2019-02-23', '%Y-%m-%d')\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: arrival_record
partitions: NULL
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 109501508
filtered: 33.33
Extra: Using where
1 row in set, 1 warning (0.00 sec)

The delete statement does not use the index (no suitable index is available), taking full table scan, resulting in a long execution time

Optimization is the establishment of a separate index indx_receive_time (receive_time)

test

Copy arrival_record test case table to delete the reindex operation
XX instance table information arrival_record

du -sh /datas/mysql/data/3316/cq_new_cimiss/arrival_record*
12K /datas/mysql/data/3316/cq_new_cimiss/arrival_record.frm
48G /datas/mysql/data/3316/cq_new_cimiss/arrival_record.ibd

COUNT the SELECT ( ) from cq_new_cimiss. arrival_record;
+ ----------- +
| COUNT ( ) |
+ ----------- +
| 112 294 946 |
+ -------- --- +
more than 100 million record number

SELECT
table_name,
CONCAT(FORMAT(SUM(data_length) / 1024 / 1024,2),'M') AS dbdata_size,
CONCAT(FORMAT(SUM(index_length) / 1024 / 1024,2),'M') AS dbindex_size,
CONCAT(FORMAT(SUM(data_length + index_length) / 1024 / 1024 / 1024,2),'G') AS table_size(G),
AVG_ROW_LENGTH,table_rows,update_time
FROM
information_schema.tables
WHERE table_schema = 'cq_new_cimiss' and table_name='arrival_record';

+----------------+-------------+--------------+------------+----------------+------------+---------------------+
| table_name | dbdata_size | dbindex_size | table_size(G) | AVG_ROW_LENGTH | table_rows | update_time |
+----------------+-------------+--------------+------------+----------------+------------+---------------------+
| arrival_record | 18,268.02M | 13,868.05M | 31.38G | 175 | 109155053 | 2019-03-26 12:40:17 |
+----------------+-------------+--------------+------------+----------------+------------+---------------------+

Disk space 48G, mysql table in the size of 31G, 17G presence around the debris, mostly due to the deletion caused by the (record is deleted, the space is not recovered)

Backup and restore to the new instance of the table, delete the original composite index, re-add the index test

mydumper parallel compressed backup

 user=root
  passwd=xxxx
 socket=/datas/mysql/data/3316/mysqld.sock
 db=cq_new_cimiss
 table_name=arrival_record
 backupdir=/datas/dump_$table_name mkdir -p $backupdir nohup echo `date +%T` && mydumper -u $user -p $passwd -S $socket -B $db -c -T $table_name -o $backupdir -t 32 -r 2000000 && echo `date +%T` &

Parallel compression backup time spent (52s) and space (1.2G, actually the table occupied by disk space for the 48G, mydumper parallel compression backup compression ratio is quite high!)

Started dump at: 2019-03-26 12:46:04 ........ Finished dump at: 2019-03-26 12:46:56 du -sh /datas/dump_arrival_record/ 1.2G /datas/dump_arrival_record/

Copy dump data to the test node
scp -rp / datas / dump_arrival_record [email protected]:

Multithreading import data

time myloader -u root -S /datas/mysql/data/3308/mysqld.sock -P 3308 -p root -B test -d /datas/dump_arrival_record -t 32

real 126m42.885s
user 1m4.543s
sys 0m4.267s

Logical Disk space after the import table

-h -d 1 /datas/mysql/data/3308/test/arrival_record.* du
12K /datas/mysql/data/3308/test/arrival_record.frm
30G /datas/mysql/data/3308/test/arrival_record.ibd
the same size free of debris, and the mysql table

cp-Rp / datas / mysql / data / 3308 / datas

Were used online DDL and pt-osc tools to do delete the replay operations
to remove the foreign key, foreign key is not deleted, composite index can not be deleted, the foreign key column belonging to the first column of the composite index

& /tmp/ddl_index.sh the bash the nohup
2019-04-04-10: 41 is: 39 STOP mysqld_3308 the begin
2019-04-04-10: 41 is: 41 is the begin and CP RM -rf DATADIR -rp datadir_bak
2019-04-04- Start mysqld_3308 10:46:53
2019-04-04- 10:46:59 Online DDL the begin
2019-04-04- 11:20:34 onlie DDL STOP
2019-04-04-11: 20 is: 34 is the begin STOP mysqld_3308
2019 -04-04-11: 20 is: 36 RM -rf DATADIR the begin and CP -rp datadir_bak
2019-04-04-11: 22 is: 48 Start mysqld_3308
2019-04-04- 11:22:53 Pt OSC the begin-
2019- 04-04- 12:19:15 pt-osc STOP
Online DDL takes time of 34 minutes, pt-osc takes time 57 minutes using onlne ddl time is about half the time the tool pt-osc

Do DDL reference

Implement

Because it is a master-slave instance, vip application is connected, the index rebuild deleted using online ddl do. Stopping the primary copy from the first doing (without recording the binlog) from the example, the main switch, then do (without recording the binlog) a new instance of the switching from the

function red_echo () {

        local what="$*" echo -e "$(date +%F-%T) ${what}" } function check_las_comm(){ if [ "$1" != "0" ];then red_echo "$2" echo "exit 1" exit 1 fi } red_echo "stop slave" mysql -uroot -p$passwd --socket=/datas/mysql/data/${port}/mysqld.sock -e"stop slave" check_las_comm "$?" "stop slave failed" red_echo "online ddl begin" mysql -uroot -p$passwd --socket=/datas/mysql/data/${port}/mysqld.sock -e"set sql_log_bin=0;select now() as ddl_start;ALTER TABLE $db_.\`${table_name}\` DROP FOREIGN KEY FK_arrival_record_product,drop index IXFK_arrival_record,add index idx_product_id_sequence_station_no(product_id,sequence,station_no),add index idx_receive_time(receive_time);select now() as ddl_stop" >>${log_file} 2>& 1 red_echo "onlie ddl stop" red_echo "add foreign key" mysql -uroot -p$passwd --socket=/datas/mysql/data/${port}/mysqld.sock -e"set sql_log_bin=0;ALTER TABLE $db_.${table_name} ADD CONSTRAINT _FK_${table_name}_product FOREIGN KEY (product_id) REFERENCES cq_new_cimiss.product (id) ON DELETE NO ACTION ON UPDATE NO ACTION;" >>${log_file} 2>& 1 check_las_comm "$?" "add foreign key error" red_echo "add foreign key stop" red_echo "start slave" mysql -uroot -p$passwd --socket=/datas/mysql/data/${port}/mysqld.sock -e"start slave" check_las_comm "$?" "start slave failed"

execution time

2019-04-08-11:. 17: 36 STOP Slave
MySQL: [Warning] A password ON the Using The Command Line interface CAN BE in the insecure.
Ddl_start
2019-04-08 11:17:36
ddl_stop
2019-04-08 11:45 : 13 is
2019-04-08-11: 45: 13 is onlie DDL STOP
2019-04-08- 11:45:13 the Add Foreign Key
MySQL: [Warning] A password ON the Using The Command Line interface CAN BE in the insecure.
2019-04 -08-12: 33 is: the Add foreign key 48 STOP
2019-04-08- 12:33:48 Start Slave
delete the index rebuild takes time of 28 minutes, add a foreign key constraint time of 48 minutes

View the execution plan delete and select statements again

explain select count(*) from arrival_record where receive_time < STR_TO_DATE('2019-03-10', '%Y-%m-%d')\G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: arrival_record
partitions: NULL
type: range
possible_keys: idx_receive_time
key: idx_receive_time
key_len: 6
ref: NULL
rows: 7540948
filtered: 100.00
Extra: Using where; Using index

explain select count(*) from arrival_record where product_id=26 and receive_time between '2019-03-25 14:00:00' and '2019-03-25 15:00:00' and receive_spend_ms>=0\G;
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: arrival_record
partitions: NULL
type: range
possible_keys: idx_product_id_sequence_station_no,idx_receive_time
key: idx_receive_time
key_len: 6
ref: NULL
rows: 291448
filtered: 16.66
Extra: Using index condition; Using where
都使用到了idx_receive_time 索引，扫描的行数大大降低

After the Index Tuning

delete or time spent 77s

delete from arrival_record where receive_time < STR_TO_DATE('2019-03-10', '%Y-%m-%d')\G

delete statement to delete records more than 300 million indexed by receive_time spend time 77s *

delete large table is optimized for small batch delete

Has been optimized to the application side 10 minutes each data (each execution time of about 1s) deleted, xxx appears not in the SLA (delay from the master alarm)

Another method is by sequential primary key records each deletion 20000

#得到满足时间条件的最大主键ID
#通过按照主键的顺序去 顺序扫描小批量删除数据
#先执行一次以下语句
 SELECT MAX(id) INTO @need_delete_max_id FROM `arrival_record` WHERE receive_time<'2019-03-01' ; DELETE FROM arrival_record WHERE id<@need_delete_max_id LIMIT 20000; select ROW_COUNT(); #返回20000 #执行小批量delete后会返回row_count(), 删除的行数 #程序判断返回的row_count()是否为0，不为0执行以下循环，为0退出循环，删除操作完成 DELETE FROM arrival_record WHERE id<@need_delete_max_id LIMIT 20000; select ROW_COUNT(); #程序睡眠0.5s

to sum up

Table data is too big, are concerned about access to the response time of the table, but also concerned about the maintenance cost of the table (such as Table DDL do more too long, delete the historical data)
When large table DDL operations, to consider the actual situation of the table (and delivered to the table, such as whether there is a foreign key) to select the appropriate way to change DDL
Scale large data delete, delete the manner of small quantities, reduce the pressure and the primary instance of the master-slave delay