I. Introduction to Scripting
languages: bash script, awk, sed
function: After mysql binlog log format, and import the database directly load data, and then do the analysis of the binlog log by sql.
Advantages: awk through progressive analysis, efficiency still can. The reason can be found in all kinds of unexpected system by analyzing the binlog log, type tables in the usage of DDL analysis system, can be done in seconds TPS level of analysis.
After extracting binlog_analyze.tar.gz get three scripts
binlog_analyze.awk: binlog used to convert text into a log directly load data into database format
binlog_analyze.sed: binlog format text you log, remove some windows and line breaks extra tabs
run.sh: script encapsulates the above two scripts, the user directly calls.
Second, use a script
to enter this directory, execute permissions given to the script.
Specify one or more binary logs need to be converted, the results generated will be placed in the target directory, ending .sql, about the same size and binary logs.
./Run.sh ../db03-bin/db03-bin.000042 ../db03-bin/db03-bin.000043 #
the OK: ../db03-bin/db03-bin.000042 IS MySQL Replication log!
The OK :! ../db03-bin/db03-bin.000043 IS MySQL Replication log
the Format binlog ING ..................
Another use, with a * instead of the target file, a directory traversal target, target directory for each file We handle
# ./run.sh ../db03-bin/*
../db03-bin/* LL
-rwxr XR-X-27-Aug. 1 Andy Andy 16:15 1073741986 DB03-bin.000033
-rwxr XR-X-27-Aug. 1 the root 1,325,409,484 the root-DB03 bin.000033 16:51. txt.awk.sql
-rwxr XR-X-27-Aug. 1 Andy Andy 1,073,741,881 16:12 DB03-bin.000034
-rwxr XR-X-27-Aug. 1 the root 1,310,773,234 the root-DB03 bin.000034.txt.awk 17:03. sql
the generated files can be imported directly into the database.
Third, the files into the database
create database binlog_statis;
use binlog-statis;
DROP TABLE IF EXISTS bin_log;
CREATE TABLE `bin_log` (
`at_pos` INT(11) NOT NULL,
`at_time` TIMESTAMP NOT NULL ,
`server_id` INT(11) NOT NULL,
`end_log_pos` INT(11) NOT NULL,
`thread_type` VARCHAR(500) DEFAULT NULL,
`dml_type` VARCHAR(50) DEFAULT NULL,
`dml_table` VARCHAR(500) DEFAULT NULL,
`dml_sql` VARCHAR(5000) DEFAULT NULL
) ENGINE=myisam DEFAULT CHARSET=utf8;
mysql> LOAD DATA INFILE ‘/home/andy/db03-bin/db03-bin.000033.txt.awk.sql’ INTO TABLE binlog_statis.`bin_log` fields TERMINATED BY “|||||”;
Query OK, 3957099 rows affected, 700 warnings (51.89 sec)
Records: 3957099 Deleted: 0 Skipped: 0 Warnings: 700
mysql> LOAD DATA INFILE ‘/home/andy/db03-bin/db03-bin.000034.txt.awk.sql’ INTO TABLE binlog_statis.`bin_log` fields TERMINATED BY “|||||”;
Query OK, 3299350 rows affected, 1334 warnings (48.79 sec)
Records: 3299350 Deleted: 0 Skipped: 0 Warnings: 1334
warnings here can not control, mainly due to long sql statement, a period of general import data to do analysis on it, plus the index after import your own, then you can cycle a day, week, month.
After the import is complete you can begin to analyze the
# According to other monitoring, found that transactions a certain time period burst high, you can inquire DDL by a specified period of time, what causes analysis can be accurate to the second.
Here are some general analysis statements, if you can dig out through this table more meaningful data, please let me know.
# TPS calculate the number per minute of a period of time
SELECT
SUBSTR(at_time, 1, 16),
COUNT(*)
FROM
`bin_log`
WHERE dml_table <> ’0′
AND at_time > ’2012-08-18 12:00:00′
AND at_time < '2012-08-20 12:00:00'
GROUP BY SUBSTR(at_time, 1, 16)
ORDER BY SUBSTR(at_time, 1, 16) ;# View the number of execution a period of time delete, insert, update, similar to the global status inside com a few percent of output, but this time may be limited.
SELECT
dml_type,
COUNT(*)
FROM
`bin_log`
WHERE at_time > ’2012-08-18 12:00:00′
AND at_time < '2012-08-20 12:00:00'
GROUP BY dml_type ;# View a period of time, the operating frequency of each table
SELECT
dml_table,
COUNT(*)
FROM
`bin_log`
WHERE at_time > ’2012-08-18 12:00:00′
AND at_time < '2012-08-20 12:00:00'
AND dml_table <> ’0′
GROUP BY dml_table ;# View a period of time, a variety of operating conditions for a table, you can clearly see a table in the main job of the kind of DDL operations
SELECT
dml_table,
dml_type,
COUNT(*)
FROM
`bin_log`
WHERE at_time > ’2012-08-18 12:00:00′
AND at_time < '2012-08-20 12:00:00'
AND dml_table <> ’0′
GROUP BY dml_table,dml_type ;# View a table in a period of visits, a 10 minutes interval
SELECT
dml_table,
SUBSTR(at_time, 1, 15),
COUNT(*)
FROM
`bin_log`
WHERE at_time > ’2012-09-10 12:00:00′
AND at_time < '2012-09-12 12:00:00'
AND dml_table = '`promotionurlreferer`'
GROUP BY dml_table,SUBSTR(at_time, 1, 15);Fourth, the script Download
binlog_analyze-0.1.tar
Transfer from: http: //isadba.com/ p = 354?
Reproduced in: https: //my.oschina.net/766/blog/211530