SQL调优案例，MYSQL服务器性能调优

其他 2019-05-07 12:20:56 阅读次数: 0
版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/moscot_wu/article/details/78058976

服务器环境见下表

服务器环境

序号

名称

版本

内存

CPU

1

Windows Server 2008 64bit

R2

16G

2

2

Mysql

5.7

256M

2

一、问题现象

1、服务器CPU使用率不稳定，波动比较明显，偶尔飙升至100%，内存使用率稳定在10%左右。

2、MYSQL相应异常缓慢，部分查询结果等待时间极长，切无法出数据。

二、检查思路

1、CPU飙升应该是有大的查询语句在执行，使用show full processlist检查查询语句。

2、内存只用到10%，应该是安装时使用了mysql的默认参数导致，需要提升mysql的相关参数。

3、收集用户反馈检查部分表和数据是否有异常。

三、处理方法

1、首先使用show full processlist 得到占用大量CPU时间的查询语句

  4	root	test.test.com:49357	x3	Query	23	Sending data	SELECT T.R_R_ID, T.RECORD_LOCK, T.NORM_VALUE, T.URGE_PERIOD,\n\t\t   E.ENTITY_ID, E.PARENT_ID, E.NAME ENTITY_NAME, E.NOTE ENTITY_NOTE, E.TYPE_ID ENTITY_TYPE,\n\t       N.NORM_ID, N.DEF_CODE NORM_CODE, N.TEST_COND, N.CALCULATE_MATH, N.CHINESE_MATH,\n\t       D.NAME NORM_NAME, D.DATA_TYPE, CD.NAME UNIT_NAME\n\t  FROM  TEST_NORM N\n\t INNER JOIN  TEST_NORM_DEFIN D ON D.NORM_DEFIN_ID = N.NORM_DEFIN_ID\n\t INNER JOIN  TEST_ENTITY E ON E.CODE = N.ENTITY_CODE\n\t  LEFT JOIN SYS_GENERAL_CODE CD ON CD.ENCODE_VALUE = D.UNIT_NAME AND CD.KEY_VALUE LIKE 'DWMC_%'\n\t  LEFT JOIN (SELECT DISTINCT RR.*\n\t  \t\t\t  FROM (\n\t  \t\t\t   SELECT DISTINCT A.R_R_ID,A.R_U_INFO_ID,A.ENTITY_ID,A.RECORD_LOCK,A.NORM_CODE,A.NORM_VALUE,A.URGE_PERIOD \n\t  \t\t\t   FROM  TEST_REPORT_RECORD A\n\t  \t\t\t  \t\tWHERE A.URGE_PERIOD_YEAR IN (2017)\n\t\t\t\t   \t\tAND A.URGE_PERIOD_MONTH IN (9)\n\t\t\t\t   \t\tAND A.URGE_PERIOD_DAY IN (20)\n\t\t\t\t   \t\t \n\t\t\t\t   \t\tAND A.NORM_CODE IN('SJA_FDL_D_D_D_SUM_M_MANL','SJA_GDL_D_D_D_SUM_M_CTST','SJA_SCCYDL_D_D_D_SUM_M_MANL','SJA_JJDL_D_D_D_SUM_M_MANL','SJA_ZGFH_D_D_D_MAX_M_MANL','SJA_PJFH_D_D_D_PAV_M_CTST','SJA_FHLV_D_D_D_PAV_M_CTST','SJA_FHXS_D_D_D_PAV_M_CTST','SJA_HYM_D_D_D_SUM_M_MANL','SJA_HYY_D_D_D_SUM_M_MANL','SJA_FDZJZB_D_D_D_SUM_M_MANL','SJA_FDYMH_D_D_D_SUM_M_CTST','SJA_FDRYH_D_D_D_SUM_M_CTST','SJA_FDBZMHZ_D_D_D_PAV_M_CTST','SJA_GDBZMHZ_D_D_D_PAV_M_CTST','SJA_MPRL_D_D_D_SUM_M_CTST','SJA_KCM_D_D_D_SIN_M_MANL','SJA_HLKC_D_D_D_SUM_M_CTST','SJA_JJKCM_D_D_D_SUM_M_CTST','SJA_MFHKYTS_D_D_D_SUM_M_MANL','SJA_LYXS_D_D_D_SUM_M_CTST','SJA_YXXS_D_D_D_PAV_M_CSST','SJA_KDCL_D_D_D_PAV_M_MANL','SJA_CLXS_D_D_D_PAV_M_CTST','SJA_JZYXQK_D_D_D_SIN_M_MANL','SJA_RBTBR_D_D_D_SIN_M_MANL','SJA_RBTJFZR_D_D_D_SIN_M_MANL')\n\t\t\t\t   \t\t \n\t\t\t\t   \t\n\t\t\t         \n\t\t\t         \n\t\t\t         \n\t\t\t         \n\t\t\t        UNION ALL(\n\t\t\t        \tSELECT DISTINCT A.R_R_ID,A.R_U_INFO_ID,A.ENTITY_ID,A.RECORD_LOCK,A.NORM_CODE,A.NORM_VALUE,A.URGE_PERIOD \n\t  \t\t\t   \t\tFROM  TEST_REPORT_RECORD A\n\t\t\t\t   \t\tWHERE A.NORM_CODE IN('SJA_FDL_D_D_D_SUM_M_MANL','SJA_ZGFH_D_D_D_MAX_M_MANL','SJA_KCM_D_D_D_SIN_M_MANL')\n\t\t\t\t   \t\tAND A.URGE_PERIOD_YEAR IN(2017)\n\t\t\t\t   \t\tAND A.URGE_PERIOD_MONTH IN(9)\n\t\t\t\t   \t\tAND A.URGE_PERIOD_DAY IN(20)\n\t\t\t        ) \n\t\t\t         \t\t\t        \t\t\t        \t\t\t        \n\t\t\t        ) RR\n\t\t\t\t) T ON N.DEF_CODE = T.NORM_CODE\n\t WHERE 1 = 1\n\t \n\t\tAND N.DEF_CODE IN('SJA_FDL_D_D_D_SUM_M_MANL','SJA_GDL_D_D_D_SUM_M_CTST','SJA_SCCYDL_D_D_D_SUM_M_MANL','SJA_JJDL_D_D_D_SUM_M_MANL','SJA_ZGFH_D_D_D_MAX_M_MANL','SJA_PJFH_D_D_D_PAV_M_CTST','SJA_FHLV_D_D_D_PAV_M_CTST','SJA_FHXS_D_D_D_PAV_M_CTST','SJA_HYM_D_D_D_SUM_M_MANL','SJA_HYY_D_D_D_SUM_M_MANL','SJA_FDZJZB_D_D_D_SUM_M_MANL','SJA_FDYMH_D_D_D_SUM_M_CTST','SJA_FDRYH_D_D_D_SUM_M_CTST','SJA_FDBZMHZ_D_D_D_PAV_M_CTST','SJA_GDBZMHZ_D_D_D_PAV_M_CTST','SJA_MPRL_D_D_D_SUM_M_CTST','SJA_KCM_D_D_D_SIN_M_MANL','SJA_HLKC_D_D_D_SUM_M_CTST','SJA_JJKCM_D_D_D_SUM_M_CTST','SJA_MFHKYTS_D_D_D_SUM_M_MANL','SJA_LYXS_D_D_D_SUM_M_CTST','SJA_YXXS_D_D_D_PAV_M_CSST','SJA_KDCL_D_D_D_PAV_M_MANL','SJA_CLXS_D_D_D_PAV_M_CTST','SJA_JZYXQK_D_D_D_SIN_M_MANL','SJA_RBTBR_D_D_D_SIN_M_MANL','SJA_RBTJFZR_D_D_D_SIN_M_MANL')\n\t \n\t \n\t \n    \t  AND '20170920' >= DATE_FORMAT(E.ACTIVE_START_TIME, '%Y%m%d') AND '20170920' <= DATE_FORMAT(E.ACTIVE_END_TIME, '%Y%m%d')  \n     \n\t \n       \tAND E.ENTITY_ID IN ('0202010110')\n     \n     \n     ORDER BY E.SORT, E.ENTITY_ID, FIND_IN_SET(SUBSTR(N.DEF_CODE, INSTR(N.DEF_CODE, '_') + 1), 'FDL_D_D_D_SUM_M_MANL,GDL_D_D_D_SUM_M_CTST,SCCYDL_D_D_D_SUM_M_MANL,JJDL_D_D_D_SUM_M_MANL,ZGFH_D_D_D_MAX_M_MANL,PJFH_D_D_D_PAV_M_CTST,FHLV_D_D_D_PAV_M_CTST,FHXS_D_D_D_PAV_M_CTST,HYM_D_D_D_SUM_M_MANL,HYY_D_D_D_SUM_M_MANL,FDZJZB_D_D_D_SUM_M_MANL,FDYMH_D_D_D_SUM_M_CTST,FDRYH_D_D_D_SUM_M_CTST,FDBZMHZ_D_D_D_PAV_M_CTST,GDBZMHZ_D_D_D_PAV_M_CTST,MPRL_D_D_D_SUM_M_CTST,KCM_D_D_D_SIN_M_MANL,HLKC_D_D_D_SUM_M_CTST,JJKCM_D_D_D_SUM_M_CTST,MFHKYTS_D_D_D_SUM_M_MANL,LYXS_D_D_D_SUM_M_CTST,YXXS_D_D_D_PAV_M_CSST,KDCL_D_D_D_PAV_M_MANL,CLXS_D_D_D_PAV_M_CTST,JZYXQK_D_D_D_SIN_M_MANL,RBTBR_D_D_D_SIN_M_MANL,RBTJFZR_D_D_D_SIN_M_MANL'), T.URGE_PERIOD

2、使用文本工具处理后结果如下 

SELECT T.R_R_ID,
       T.RECORD_LOCK,
       T.NORM_VALUE,
       T.URGE_PERIOD,
       E.ENTITY_ID,
       E.PARENT_ID,
       E.NAME ENTITY_NAME,
       E.NOTE ENTITY_NOTE,
       E.TYPE_ID ENTITY_TYPE,
       N.NORM_ID,
       N.DEF_CODE NORM_CODE,
       N.TEST_COND,
       N.CALCULATE_MATH,
       N.CHINESE_MATH,
       D.NAME NORM_NAME,
       D.DATA_TYPE,
       CD.NAME UNIT_NAME
  FROM  TEST_NORM N
 INNER JOIN  TEST_NORM_DEFIN D ON D.NORM_DEFIN_ID = N.NORM_DEFIN_ID
 INNER JOIN  TEST_ENTITY E ON E.CODE = N.ENTITY_CODE
  LEFT JOIN SYS_GENERAL_CODE CD ON CD.ENCODE_VALUE = D.UNIT_NAME
                               AND CD.KEY_VALUE LIKE 'DWMC_%'
  LEFT JOIN (SELECT DISTINCT RR.*
               FROM (SELECT DISTINCT A.R_R_ID,
                                     A.R_U_INFO_ID,
                                     A.ENTITY_ID,
                                     A.RECORD_LOCK,
                                     A.NORM_CODE,
                                     A.NORM_VALUE,
                                     A.URGE_PERIOD
                       FROM  TEST_REPORT_RECORD A
                      WHERE A.URGE_PERIOD_YEAR IN (2017)
                        AND A.URGE_PERIOD_MONTH IN (9)
                        AND A.URGE_PERIOD_DAY IN (20)
                        AND A.NORM_CODE IN ('SJA_FDL_D_D_D_SUM_M_MANL',
                             'SJA_GDL_D_D_D_SUM_M_CTST',
                             'SJA_SCCYDL_D_D_D_SUM_M_MANL',
                             'SJA_JJDL_D_D_D_SUM_M_MANL',
                             'SJA_ZGFH_D_D_D_MAX_M_MANL',
                             'SJA_PJFH_D_D_D_PAV_M_CTST',
                             'SJA_FHLV_D_D_D_PAV_M_CTST',
                             'SJA_FHXS_D_D_D_PAV_M_CTST',
                             'SJA_HYM_D_D_D_SUM_M_MANL',
                             'SJA_HYY_D_D_D_SUM_M_MANL',
                             'SJA_FDZJZB_D_D_D_SUM_M_MANL',
                             'SJA_FDYMH_D_D_D_SUM_M_CTST',
                             'SJA_FDRYH_D_D_D_SUM_M_CTST',
                             'SJA_FDBZMHZ_D_D_D_PAV_M_CTST',
                             'SJA_GDBZMHZ_D_D_D_PAV_M_CTST',
                             'SJA_MPRL_D_D_D_SUM_M_CTST',
                             'SJA_KCM_D_D_D_SIN_M_MANL',
                             'SJA_HLKC_D_D_D_SUM_M_CTST',
                             'SJA_JJKCM_D_D_D_SUM_M_CTST',
                             'SJA_MFHKYTS_D_D_D_SUM_M_MANL',
                             'SJA_LYXS_D_D_D_SUM_M_CTST',
                             'SJA_YXXS_D_D_D_PAV_M_CSST',
                             'SJA_KDCL_D_D_D_PAV_M_MANL',
                             'SJA_CLXS_D_D_D_PAV_M_CTST',
                             'SJA_JZYXQK_D_D_D_SIN_M_MANL',
                             'SJA_RBTBR_D_D_D_SIN_M_MANL',
                             'SJA_RBTJFZR_D_D_D_SIN_M_MANL')
                     UNION ALL (SELECT DISTINCT A.R_R_ID,
                                               A.R_U_INFO_ID,
                                               A.ENTITY_ID,
                                               A.RECORD_LOCK,
                                               A.NORM_CODE,
                                               A.NORM_VALUE,
                                               A.URGE_PERIOD
                                 FROM  TEST_REPORT_RECORD A
                                WHERE A.NORM_CODE IN
                                      ('SJA_FDL_D_D_D_SUM_M_MANL',
                                       'SJA_ZGFH_D_D_D_MAX_M_MANL',
                                       'SJA_KCM_D_D_D_SIN_M_MANL')
                                  AND A.URGE_PERIOD_YEAR IN (2017)
                                  AND A.URGE_PERIOD_MONTH IN (9)
                                  AND A.URGE_PERIOD_DAY IN (20))) RR) T ON N.DEF_CODE =
                                                                           T.NORM_CODE
 WHERE 1 = 1
   AND N.DEF_CODE IN
       ('SJA_FDL_D_D_D_SUM_M_MANL', 'SJA_GDL_D_D_D_SUM_M_CTST',
        'SJA_SCCYDL_D_D_D_SUM_M_MANL', 'SJA_JJDL_D_D_D_SUM_M_MANL',
        'SJA_ZGFH_D_D_D_MAX_M_MANL', 'SJA_PJFH_D_D_D_PAV_M_CTST',
        'SJA_FHLV_D_D_D_PAV_M_CTST', 'SJA_FHXS_D_D_D_PAV_M_CTST',
        'SJA_HYM_D_D_D_SUM_M_MANL', 'SJA_HYY_D_D_D_SUM_M_MANL',
        'SJA_FDZJZB_D_D_D_SUM_M_MANL', 'SJA_FDYMH_D_D_D_SUM_M_CTST',
        'SJA_FDRYH_D_D_D_SUM_M_CTST', 'SJA_FDBZMHZ_D_D_D_PAV_M_CTST',
        'SJA_GDBZMHZ_D_D_D_PAV_M_CTST', 'SJA_MPRL_D_D_D_SUM_M_CTST',
        'SJA_KCM_D_D_D_SIN_M_MANL', 'SJA_HLKC_D_D_D_SUM_M_CTST',
        'SJA_JJKCM_D_D_D_SUM_M_CTST', 'SJA_MFHKYTS_D_D_D_SUM_M_MANL',
        'SJA_LYXS_D_D_D_SUM_M_CTST', 'SJA_YXXS_D_D_D_PAV_M_CSST',
        'SJA_KDCL_D_D_D_PAV_M_MANL', 'SJA_CLXS_D_D_D_PAV_M_CTST',
        'SJA_JZYXQK_D_D_D_SIN_M_MANL', 'SJA_RBTBR_D_D_D_SIN_M_MANL',
        'SJA_RBTJFZR_D_D_D_SIN_M_MANL')
   AND '20170920' >= DATE_FORMAT(E.ACTIVE_START_TIME, '%Y%m%d')
   AND '20170920' <= DATE_FORMAT(E.ACTIVE_END_TIME, '%Y%m%d')
   AND E.ENTITY_ID IN ('0202010110')
 ORDER BY E.SORT,
          E.ENTITY_ID,
          FIND_IN_SET(SUBSTR(N.DEF_CODE, INSTR(N.DEF_CODE, '_') + 1),
                      'FDL_D_D_D_SUM_M_MANL,GDL_D_D_D_SUM_M_CTST,SCCYDL_D_D_D_SUM_M_MANL,JJDL_D_D_D_SUM_M_MANL,ZGFH_D_D_D_MAX_M_MANL,PJFH_D_D_D_PAV_M_CTST,FHLV_D_D_D_PAV_M_CTST,FHXS_D_D_D_PAV_M_CTST,HYM_D_D_D_SUM_M_MANL,HYY_D_D_D_SUM_M_MANL,FDZJZB_D_D_D_SUM_M_MANL,FDYMH_D_D_D_SUM_M_CTST,FDRYH_D_D_D_SUM_M_CTST,FDBZMHZ_D_D_D_PAV_M_CTST,GDBZMHZ_D_D_D_PAV_M_CTST,MPRL_D_D_D_SUM_M_CTST,KCM_D_D_D_SIN_M_MANL,HLKC_D_D_D_SUM_M_CTST,JJKCM_D_D_D_SUM_M_CTST,MFHKYTS_D_D_D_SUM_M_MANL,LYXS_D_D_D_SUM_M_CTST,YXXS_D_D_D_PAV_M_CSST,KDCL_D_D_D_PAV_M_MANL,CLXS_D_D_D_PAV_M_CTST,JZYXQK_D_D_D_SIN_M_MANL,RBTBR_D_D_D_SIN_M_MANL,RBTJFZR_D_D_D_SIN_M_MANL'),
          T.URGE_PERIOD

注该语句查询经过索引但是查询时间还是高达30多秒。

4、通过应用管理员反馈一条简单的SQL查询也很久无法相应，同样该查询走了索引，用时36.7s

select *
  from test_report_record r
 where r.NORM_CODE = 'SJC2_CJX_D_D_D_SUM_M_MANL'
   and r.URGE_PERIOD_YEAR = 2017
   and r.URGE_PERIOD_MONTH = 9
   and r.URGE_PERIOD_DAY = 20
test_report_record 该表记录数比较大，为了得到该表的数据进行了count操作 

select count(1) from test_report_record.
但是无法count出结果，应该是test_report_record表出现问题。且经过查询发现，该表数据量较大并未做过分区表，根据应用需求该表设计为分区表比较合理。

5、新建一张test_report_record2分区表，将test_report_record表的数据导入一份到该表，使用相同语句进行查询确认是否相应速度得到提升。

CREATE TABLE `test_report_record2` (
  `R_R_ID` varchar(32) NOT NULL DEFAULT '',
  `NORM_VALUE` varchar(2000) DEFAULT NULL ,
  `INPUT_VALUE` varchar(2000) DEFAULT NULL,
  `NORM_CODE` varchar(64) NOT NULL DEFAULT '',
  `R_U_INFO_ID` varchar(32) DEFAULT NULL ,
  `PUSH_TIME` datetime DEFAULT NULL ,
  `STATE` int(11) DEFAULT NULL ,
  `SYS_USER_ID` varchar(32) DEFAULT NULL,
  `URGE_PERIOD_YEAR` int(11) ,
  `URGE_PERIOD_MONTH` int(11) DEFAULT NULL,
  `URGE_PERIOD_DAY` int(11) DEFAULT NULL,
  `URGE_PERIOD` date DEFAULT NULL,
  `ENTITY_ID` varchar(64) DEFAULT NULL ,
  `RECORD_LOCK` int(4) DEFAULT NULL ,
  `SEND` int(11) DEFAULT '0'  ,
  PRIMARY KEY (`R_R_ID`,`URGE_PERIOD_YEAR`),
  KEY `R_U_INFO_ID2` (`R_U_INFO_ID`) USING BTREE,
  KEY `NORM_CODE2` (`NORM_CODE`) USING BTREE,
  KEY `URGE_PERIOD2` (`URGE_PERIOD`) USING BTREE,
  KEY `URGE2` (`URGE_PERIOD_YEAR`,`URGE_PERIOD_MONTH`,`URGE_PERIOD_DAY`) USING BTREE,
  KEY `PUSH_TIME2` (`PUSH_TIME`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8  
    PARTITION BY RANGE(URGE_PERIOD_YEAR) (
    PARTITION p_2015 VALUES LESS THAN (2015),
    PARTITION p_2016 VALUES LESS THAN (2016),
    PARTITION p_2017 VALUES LESS THAN (2017),
    PARTITION p_2018 VALUES LESS THAN (2018),
    PARTITION p_2019 VALUES LESS THAN (2019),
    PARTITION p_2020 VALUES LESS THAN (2020),
    PARTITION p_2021 VALUES LESS THAN (2021),
    PARTITION p_2022 VALUES LESS THAN (2022),
    PARTITION p_2023 VALUES LESS THAN (2023),
    PARTITION p_2024 VALUES LESS THAN (2024),
    PARTITION p_2025 VALUES LESS THAN (2025),
    PARTITION p_catchall VALUES LESS THAN MAXVALUE);

6、停止mysql数据库停止应用。检查发现Mysql使用的是默认的参数，针对这种情况对相关参数进行了调整

myisam_sort_buffer_size=128M
key_buffer_size=2048M 
read_buffer_size=64M
read_rnd_buffer_size=64M
sort_buffer_size=16M
innodb_log_buffer_size=16M
innodb_buffer_pool_size=2048M
innodb_log_file_size=128M
max_allowed_packet=32M

7、启动MYSQL，不启动应用。统计test_report_record数据，依旧无法得到结果，对该表进行大的查询时候会报1205的错误。


8、确认test_report_record该表统计信息有问题（oracle经验），决定对该表进行统计信息的收集和优化工作

mysql> analyze table test_report_record
9、分析完毕改表后，重新对test_report_record进行统计，结果如下664w行用了73s


10、使用如下方法插入并将test_report_record该表改成分区表

insert into test_report_record2 select * from test_report_record;
alter table test_report_record rename test_report_recordbak;
alter table test_report_record2 rename test_report_record;

11、验证结果现在查询仅需要0.008s就有结果，比之前快了几个数量级，目前的执行计划也走了分区


12、打开应用，检查报表，访问速度均得到较大的提升，优化完毕。

四、问题总结

随着系统的使用，应用数据会越来越多，应用在设计之初的性能问题就会慢慢表现出来，此时维护的重点就需要对系统进行优化。

1、mysql安装时初始参数一般都较小，可以根据服务器的配置进行提升，上面的参数仅供参考，实际上还可设置大点。

2、mysql的表的信息收集机制和oracle还是有比较大的不同，最近发现在oracle本来不会成为问题的在Mysql上运行会存在巨大的性能问题，再出现表无法及时返回时注意使用optimize和analyze对表进行分析和优化。

3、对于部分预计数据量很大的表可以考虑使用分区表。

4、不管是什么类型的数据库都要注意对数据进行备份，防止表数据意外丢失或者坏道时可以保证数据恢复。

猜你喜欢

转载自blog.csdn.net/moscot_wu/article/details/78058976
今日推荐
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
周排行
Java自定义时间格式
同步整形电路
在开发中最最最常用的字符串的属性大集合
Linux 查看端口占用并杀掉
Java基础四：ArrayList
多线程之死锁就是这么简单
mysql 基础命令集
awk 命令详解
Centos6.3编译安装nginx+php步骤
OCR （Optical Character Recognition，光学字符识别）
每日归档
更多
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022