1. Introduction to SQLAdvisor

SQLAdvisor is a SQL optimization tool developed and maintained by the Beijing DBA team of Meituan Dianping Company: input SQL and output index optimization suggestions. It is based on MySQL's native lexical parsing, combined with where conditions in SQL, field selectivity, aggregation conditions, multi-table Join relationships, etc., and finally outputs optimal index optimization suggestions. At present, SQLAdvisor is widely used within Meituan and is relatively mature and stable.

Advantages of SQLAdvisor

Based on MySQL native lexical parsing, the performance, accuracy and stability of lexical parsing are fully guaranteed;
Support common SQL (Insert/Delete/Update/Select);
Support multi-table Join and automatically logically select the driver table;
Support aggregation conditions Order by and Group by;
Filter indexes that already exist in the table.

2. Principle of SQLAdvisor

SQLAdvisor architecture flowchart:

SQLAdvisor includes the following processing methods: Join processing, where processing, calculating discrimination, adding alternative indexes, Group and Order processing, driving table selection, adding alternative indexes for driven tables, and outputting suggestions. For the specific flow chart, please refer to: Meituan-Dianping SQL optimization tool SQLAdvisor open source

3. SQLAdvisor test

3.1 SQLAdvisor installation

3.1.1 拉取最新代码
git clone https://github.com/Meituan-Dianping/SQLAdvisor.git

3.1.2 安装依赖项
yum install -y cmake libaio-devel libffi-devel glib2 glib2-devel bison

# 因 yum 安装 Percona-Server-shared-56 失败，故使用 rpm 包安装，\
# 具体参考 https://github.com/Meituan-Dianping/SQLAdvisor/issues/12
yum install -y --enablerepo=Percona56 Percona-Server-shared-56
yum install -y Percona-Server-server-56 Percona-Server-client-56

rpm -ivh Percona-Server-shared-56-5.6.25-rel73.1.el6.x86_64.rpm

# 设置软链
cd /usr/lib64/
ls -l libperconaserverclient_r.so.18
ln -s libperconaserverclient_r.so.18 libperconaserverclient_r.so

3.1.3 编译依赖项 sqlparser
cmake -DBUILD_CONFIG=mysql_release -DCMAKE_BUILD_TYPE=debug \
-DCMAKE_INSTALL_PREFIX=/usr/local/sqlparser ./
make && make install

3.1.4 安装 SQLAdvisor 源码
cd sqladvisor/
cmake -DCMAKE_BUILD_TYPE=debug ./
make
cp sqladvisor /usr/local/bin
sqladvisor --help
Usage:
  sqladvisor [OPTION...] sqladvisor

SQL Advisor Summary

Help Options:
  -?, --help              Show help options

Application Options:
  -f, --defaults-file     sqls file
  -u, --username          username
  -p, --password          password
  -P, --port              port
  -h, --host              host
  -d, --dbname            database name
  -q, --sqls              sqls
  -v, --verbose           1:output logs 0:output nothing

3.2 Import test data

For privacy reasons, online table names are masked and replaced with tableA and tableB. The table structure of the desensitization treatment is as follows:

CREATE TABLE `tableA` ( \
  `ID` int(10) unsigned NOT NULL AUTO_INCREMENT, \
  `CATE` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `E_ID` char(18) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `RD` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `RU` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `MO` int(10) NOT NULL DEFAULT '0' COMMENT 'xxxx', \
  `LID` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `LEVEL` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'xxxx', \
  `GI` varchar(15) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `GT` datetime DEFAULT NULL COMMENT 'xxxx', \
  `CL` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `ST` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'xxxx', \
  `RES` varchar(64) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  PRIMARY KEY (`ID`), \
  UNIQUE KEY `i_e_id` (`E_ID`), \
  KEY `i_lid` (`LID`), \
  KEY `i_rd_ru_mo` (`RD`,`RU`,`MO`) \
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='tableA'

CREATE TABLE `tableB` ( \
  `ID` int(11) NOT NULL AUTO_INCREMENT, \
  `LID` varchar(32) NOT NULL COMMENT 'xxxx', \
  `NAME` varchar(45) NOT NULL COMMENT 'xxxx', \
  `CL` varchar(30) NOT NULL COMMENT 'xxxx', \
  `TIME` datetime DEFAULT NULL COMMENT 'xxxx', \
  `NUM` varchar(64) NOT NULL COMMENT 'xxxx', \
  `SOUR` varchar(32) NOT NULL COMMENT 'xxxx', \
  `GI` varchar(15) NOT NULL COMMENT 'xxxx', \
  `TYPE` tinyint(4) NOT NULL COMMENT 'xxxx', \
  `SID` int(11) NOT NULL COMMENT 'xxxx', \
  `SEID` int(11) NOT NULL COMMENT 'xxxx', \
  `NAME` varchar(20) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `ADD` varchar(255) NOT NULL COMMENT 'xxxx', \
  `PO` varchar(11) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `QQ` varchar(20) NOT NULL COMMENT 'xxxx', \
  `CATE` varchar(20) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `RU` varchar(32) NOT NULL DEFAULT '' COMMENT 'xxxx', \
  `SC` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'xxxx', \
  `LEVEL` varchar(32) DEFAULT '' COMMENT 'xxxx', \
  PRIMARY KEY (`ID`), \
  KEY `i_user` (`LID`,`CATE`), \
  KEY `i_num` (`NUM`), \
  KEY `i_cl` (`CL`) \
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='tableB'

The amount of imported data is as follows:

mysql> SELECT COUNT(*) FROM tableA;
+----------+
| COUNT(*) |
+----------+
|   122658 |
+----------+
1 row in set (0.06 sec)

mysql> SELECT COUNT(*) FROM tableB;
+----------+
| COUNT(*) |
+----------+
|   979525 |
+----------+
1 row in set (0.23 sec)

3.3 Execute the test

Test statement:

/usr/local/bin/sqladvisor -u root -p 'xxxx' -P 3306 -h xxx.xxx.xxx.xxx -d databaseA \
-q "SELECT * FROM tableA WHERE LID = 'xxxx' ORDER BY GT DESC" -v 1

Test Results

2017-03-14 12:30:51 1923 [Note] 第1步: 对SQL解析优化之后得到的SQL:\
select `*` AS `*` from `databaseA`.`tableA` where (`LID` = 'xxxx') \
order by `GT` desc
2017-03-14 12:30:51 1923 [Note] 第2步：开始解析where中的条件:(`LID` = 'xxxx')
2017-03-14 12:30:51 1923 [Note] show index from tableA
2017-03-14 12:30:51 1923 [Note] show table ST like 'tableA'
2017-03-14 12:30:51 1923 [Note] select count(*) from ( select `LID` from `tableA` \
    FORCE INDEX( i_E_ID ) order by E_ID DESC limit 10000) `tableA` \
    where (`LID` = 'xxxx')
2017-03-14 12:30:51 1923 [Note] 第3步：表tableA的行数:122879,limit行数:10000,\
得到where条件中(`LID` = 'xxxx')的选择度:10000
2017-03-14 12:30:51 1923 [Note] 第4步：开始解析order by 条件
2017-03-14 12:30:51 1923 [Note] 第5步：开始验证 字段GT是不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] show index from tableA where Key_name = 'PRIMARY' \
and Column_name ='GT' and Seq_in_index = 1
2017-03-14 12:30:51 1923 [Note] 第6步：字段GT不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] 第7步：开始添加order by 字段
2017-03-14 12:30:51 1923 [Note] 第8步：开始验证 字段GT是不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] show index from tableA where Key_name = 'PRIMARY' \
and Column_name ='GT' and Seq_in_index = 1
2017-03-14 12:30:51 1923 [Note] 第9步：字段GT不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] 第10步：开始验证 字段LID是不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] show index from tableA where Key_name = 'PRIMARY' \
and Column_name ='LID' and Seq_in_index = 1
2017-03-14 12:30:51 1923 [Note] 第11步：字段LID不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] 第12步：开始验证 字段LID是不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] show index from tableA where Key_name = 'PRIMARY' \
and Column_name ='LID' and Seq_in_index = 1
2017-03-14 12:30:51 1923 [Note] 第13步：字段LID不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] 第14步：开始验证表中是否已存在相关索引。表名:tableA, \
字段名:LID, 在索引中的位置:1
2017-03-14 12:30:51 1923 [Note] show index from tableA where Column_name ='LID' \
and Seq_in_index =1
2017-03-14 12:30:51 1923 [Note] 第15步：开始验证 字段GT是不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] show index from tableA where Key_name = 'PRIMARY' \
and Column_name ='GT' and Seq_in_index = 1
2017-03-14 12:30:51 1923 [Note] 第16步：字段GT不是主键。表名:tableA
2017-03-14 12:30:51 1923 [Note] 第17步：开始验证表中是否已存在相关索引。\
表名:tableA, 字段名:GT, 在索引中的位置:2
2017-03-14 12:30:51 1923 [Note] show index from tableA where \
Column_name ='GT' and Seq_in_index =2
2017-03-14 12:30:52 1923 [Note] 第18步：开始输出表tableA索引优化建议:
2017-03-14 12:30:52 1923 [Note] Create_Index_SQL：alter table tableA add index \
idx_LID_GT(LID,GT)
2017-03-14 12:30:52 1923 [Note] 第19步: SQLAdvisor结束!

4. Conclusion

Meituan-Dianping's open source SQL optimization tool SQLAdvisor is quite satisfied with the optimization suggestions, and it is recommended to try it online for a period of time. The cost of this tool is that it requires online DB installation related dependencies. If it is confirmed to be adopted, you can consider deploying this tool when initializing the DB server.

Original address: https://dbarobin.com/2017/03/16/test-report-of-sqladvisor/

Official address: https://github.com/Meituan-Dianping/SQLAdvisor

In order to facilitate everyone to communicate, I have opened a WeChat public account and a QQ group, QQ group: 291519319, let’s communicate with those who like technology

Meituan Dianping open source SQL optimization tool SQLAdvisor test report