Mysql bin log学习
1、mysql 版本
$ mysql -v
Server version: 5.6.41 MySQL Community Server (GPL)
2、mysql bin log 介绍
https://dev.mysql.com/doc/internals/en/binary-log.html
https://blog.csdn.net/wudongxu/article/details/6598562
2.1是否启用binlog日志
show variables like 'log_bin';
2.2查看binlog
show binlog events;
2.3查看最新一个binlog日志文件名称
show master status;
2.4查找binlog日志
find / -name mysql-bin -type f
2.5查看详细的日志配置信息
SHOW GLOBAL VARIABLES LIKE '%log%';
2.6mysql数据存储目录
show variables like '%dir%';
2.7查看binlog的目录
mysql> show global variables like "%log_bin%";
+---------------------------------+--------------------------------+
| Variable_name | Value |
+---------------------------------+--------------------------------+
| log_bin | ON |
| log_bin_basename | /var/lib/mysql/mysql-bin |
| log_bin_index | /var/lib/mysql/mysql-bin.index |
| log_bin_trust_function_creators | OFF |
| log_bin_use_v1_row_events | OFF |
| sql_log_bin | ON |
+---------------------------------+--------------------------------+
注：参数说明
https://dev.mysql.com/doc/refman/5.6/en/replication-options-binary-log.html#sysvar_log_bin_basename
1)log_bin_basename：保存二进制日志文件的基本名称和路径
2)log_bin_index；保存二进制日志索引文件的基本名称和路径，该log_bin_index系统变量在MySQL 5.6.4添加
2.8查看bin-log二进制文件
相关ddl和dml操作
/////////////////////////////////////////////////////////////////////////////////////////////
执行的sql语句：
mysql> create database test_binlog;
Query OK, 1 row affected (0.00 sec)
mysql> use test_binlog;
Database changed
mysql> create table test_bin_log (id int ,name varchar(30));
Query OK, 0 rows affected (0.02 sec)
mysql> insert into test_bin_log values(1,"mysql001");
Query OK, 1 row affected (0.00 sec)
mysql> insert into test_bin_log values(2,"mysql002");
Query OK, 1 row affected (0.01 sec)
//////////////////////////////////////////////////////////////////////////////////////////////
2.8.1方法1（推荐）
show binlog events in 'mysql-bin.000003';
mysql> show binlog events in 'mysql-bin.000003';
+------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
----------------+
| mysql-bin.000003 | 4 | Format_desc | 1 | 120 | Server ver: 5.6.41-log, Binlog ver: 4 |
| mysql-bin.000003 | 120 | Query | 1 | 235 | create database test_binlog |
| mysql-bin.000003 | 235 | Query | 1 | 375 | use `test_binlog`; create table test_bin_log (id int ,name varchar(30)) |
| mysql-bin.000003 | 375 | Query | 1 | 468 | BEGIN |
| mysql-bin.000003 | 468 | Query | 1 | 601 | use `test_binlog`; insert into test_bin_log values(1,"mysql001") |
| mysql-bin.000003 | 601 | Xid | 1 | 632 | COMMIT /* xid=69452 */ |
| mysql-bin.000003 | 632 | Query | 1 | 725 | BEGIN |
| mysql-bin.000003 | 725 | Query | 1 | 858 | use `test_binlog`; insert into test_bin_log values(2,"mysql002") |
| mysql-bin.000003 | 858 | Xid | 1 | 889 | COMMIT /* xid=69589 */ |
| mysql-bin.000003 | 889 | Query | 1 | 965 | BEGIN |
| mysql-bin.000003 | 965 | Query | 1 | 1188 | use `ambari`; UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = 'WAITING' WHERE SCHED_NAME = 'ExecutionScheduler' AND (TRIGGER_STATE = 'ACQUIRED' OR TRIGGER_STATE = 'BLOCKED') |
| mysql-bin.000003 | 1188 | Query | 1 | 1423 | use `ambari`; UPDATE QRTZ_TRIGGERS SET TRIGGER_STATE = 'PAUSED' WHERE SCHED_NAME = 'ExecutionScheduler' AND (TRIGGER_STATE = 'PAUSED_BLOCKED' OR TRIGGER_STATE = 'PAUSED_BLOCKED') |
| mysql-bin.000003 | 1423 | Query | 1 | 1572 | use `ambari`; DELETE FROM QRTZ_FIRED_TRIGGERS WHERE SCHED_NAME = 'ExecutionScheduler' |
| mysql-bin.000003 | 1572 | Query | 1 | 1647 | COMMIT |
+------------------+------+-------------+-----------+-------------2.8.2 方法2
[root@bdp03nn01 mysql]# mysqlbinlog -v --base64-output=decode-rows --start-datetime="2018-08-22 08:30:00" --stop-datetime="2018-08-26 00:30:00" /var/lib/mysql/mysql-bin.000003
\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_insert_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#180823 16:28:03 server id 1 end_log_pos 120 CRC32 0x7f57797a Start: binlog v 4, server v 5.6.41-log created 180823 16:28:03
# Warning: this binlog is either in use or was not closed properly.
BINLOG '
E3B+Ww8BAAAAdAAAAHgAAAABAAQANS42LjQxLWxvZwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAEzgNAAgAEgAEBAQEEgAAXAAEGggAAAAICAgCAAAACgoKGRkAAXp5
V38=
'/*!*/;
# at 120
#180823 16:53:40 server id 1 end_log_pos 235 CRC32 0xe6eb36b4 Query thread_id=1088 exec_time=0 error_code=0
SET TIMESTAMP=1535014420/*!*/;
SET @@session.pseudo_thread_id=1088/*!*/;
SET @@session.foreign_key_checks=1, @@session.sql_auto_is_null=0, @@session.unique_checks=1, @@session.autocommit=1/*!*/;
SET @@session.sql_mode=1075838976/*!*/;
SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/;
/*!\C utf8 *//*!*/;
SET @@session.character_set_client=33,@@session.collation_connection=33,@@session.collation_server=33/*!*/;
SET @@session.lc_time_names=0/*!*/;
SET @@session.collation_database=DEFAULT/*!*/;
create database test_binlog
/*!*/;
# at 235
#180823 16:54:38 server id 1 end_log_pos 375 CRC32 0x8267becb Query thread_id=1088 exec_time=0 error_code=0
use `test_binlog`/*!*/;
SET TIMESTAMP=1535014478/*!*/;
create table test_bin_log (id int ,name varchar(30))
/*!*/;
# at 375
#180823 16:55:12 server id 1 end_log_pos 468 CRC32 0x10fcd3fe Query thread_id=1088 exec_time=0 error_code=0
SET TIMESTAMP=1535014512/*!*/;
BEGIN
/*!*/;
# at 468
#180823 16:55:12 server id 1 end_log_pos 601 CRC32 0x161b0df5 Query thread_id=1088 exec_time=0 error_code=0
SET TIMESTAMP=1535014512/*!*/;
insert into test_bin_log values(1,"mysql001")
/*!*/;
# at 601
#180823 16:55:12 server id 1 end_log_pos 632 CRC32 0x776d3e6c Xid = 69452
COMMIT/*!*/;
# at 632
#180823 16:55:18 server id 1 end_log_pos 725 CRC32 0xc2d44757 Query thread_id=1088 exec_time=0 error_code=0
SET TIMESTAMP=1535014518/*!*/;
BEGIN
/*!*/;
# at 725
#180823 16:55:18 server id 1 end_log_pos 858 CRC32 0xde83d0d3 Query thread_id=1088 exec_time=0 error_code=0
SET TIMESTAMP=1535014518/*!*/;
insert into test_bin_log values(2,"mysql002")
/*!*/;
# at 858
#180823 16:55:18 server id 1 end_log_pos 889 CRC32 0xbb62f482 Xid = 69589
COMMIT/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
参数说明：
--start-datetime：从二进制日志中读取指定等于时间戳或者晚于本地计算机的时间
--stop-datetime： 从二进制日志中读取指定小于时间戳或者等于本地计算机的时间 取值和上述一样
--start-position：从二进制日志中读取指定position 事件位置作为开始。
--stop-position： 从二进制日志中读取指定position 事件位置作为事件截至
3、解析和采集
3.1工具
canal：
免费-阿里巴巴
（
https://github.com/alibaba/canal/wiki/Introduction，
https://slidesplayer.com/slide/11557608/，
https://docs.google.com/presentation/d/1MkszUPYRDkfVPz9IqOT1LLT5d9tuwde_WC8GZvjaDRg/edit#）
maxwells： 免费-亚马逊（http://maxwells-daemon.io/）
3.2举例
maxwells: https://github.com/bobshaw1912/cdc-kinesis-demo，https://blog.csdn.net/liguohuaBigdata/article/details/79472777
canal: https://github.com/alibaba/canal/wiki/QuickStart
3.3版本说明
3.3.1 canal
当前最高支持版本是<mysql 5.5
功能多，太重
https://github.com/alibaba/canal/wiki/AdminGuide
3.3.2 maxwells
功能少，较轻，适合我们的应用场景
4、应用场景
4.1 数据异构、缓冲和任务分发
http://www.seekwd.com/article/details/150
4.2 实现流式实时分析架构
https://aws.amazon.com/cn/blogs/china/mysql-binlog-architecture/
4.3 数据恢复
https://www.cnblogs.com/kevingrace/p/5907254.html
4.4 数据实时处理方法及系统
https://patents.google.com/patent/CN106446239A/zh
5、搭建
5.1 手把手搭建maxwell+kafka的环境
http://maxwells-daemon.io/
5.2 环境说明
5.2.1 需要的版本
apache kafka: 0.8.2以上，版本需要对应，否则可能出现不兼容
maxwells : 1.2.X以上
mysql : 5.1，5.5，5.6，5.7
jdk : 1.8以上
操作系统 : centos7
5.2.2 集群上的环境
测试集群：hdp-2.6.4.0-91、jdk-1.8.0_141、kafka-0.10.1.2.6.4.0-91
生产环境：hdp-2.6.0.3-8、jdk-1.8.0_141、kafka-0.10.1.2.6.0.3-8
自玩环境：hdp-2.6.5.0-292、jdk-1.8.0_172-b11、mysql-5.6.41、kafka-1.0.0.2.6.5.0-292
5.2.3 maxwells版本
maxwells-1.17.0 ----> https://github.com/zendesk/maxwell/tree/v1.17.0
5.3 安装
5.3.1 测试kafka是否正常安装
5.3.1.1 创建kafka的topic
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-topics.sh --create --topic maxwells --zookeeper localhost:2181 --partitions 6 --replication-factor 2
5.3.1.2 生产数据
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-producer.sh --broker-list bdp03nn01:6667,bdp03dn01:6667,bdp03nn02:6667 --topic maxwells
5.3.1.3 消费数据
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-consumer.sh --zookeeper bdp03nn01:2181,bdp03dn01:2181,bdp03nn02:2181 --topic maxwells --from-beginning
5.3.1.4 描述topic
/usr/hdp/2.6.5.0-292/kafka/bin/kafka-topics.sh --describe --zookeeper bdp03nn01:2181 --topic maxwells
5.3.2 安装maxwell
5.3.2.1 下载、并解压maxwells
$ cd /usr/local
$ wget https://github.com/zendesk/maxwell/releases/download/v1.17.0/maxwell-1.17.0.tar.gz
5.3.2.2 配置mysql
5.3.2.2.1编辑my.cnf
$ vi /etc/my.cnf
[mysqld]
server_id=1
log-bin=master
binlog_format=row
5.3.2.2.2 添加权限
mysql> GRANT ALL on *.* to 'maxwell'@'172.16.5.117' identified by 'mysqlmaxwellpwd' ;
mysql> GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'172.16.5.117';
mysql> flush privaleges;
5.3.2.2.3 刷新bin-log日志、重启mysql
mysql> flush logs;
#systemctl restart mysqld.service
5.3.3 运行maxwells
5.3.3.1 未过滤库
/usr/local/maxwell/bin/maxwell --user='maxwell' --password='mysqlmaxwellpwd' --host='172.16.5.117' \
--producer=kafka --kafka.bootstrap.servers=bdp03nn01:6667,bdp03dn01:6667,bdp03nn02:6667 \
--kafka_topic=maxwells
5.3.3.2 过滤指定规则的库
/usr/local/maxwell/bin/maxwell --user='maxwell' --password='mysqlmaxwellpwd' --host='172.16.5.117' \
--producer=kafka --kafka.bootstrap.servers=bdp03nn01:6667,bdp03dn01:6667,bdp03nn02:6667 \
--kafka_topic=maxwells --filter 'exclude: ambari.*, include: test_binlog.*'
参考过滤配置：http://maxwells-daemon.io/filtering/
5.4 消费数据
#/usr/hdp/2.6.5.0-292/kafka/bin/kafka-console-consumer.sh --zookeeper bdp03nn01:2181,bdp03dn01:2181,bdp03nn02:2181 --topic maxwells --from-beginning
命令行监控的数据：
{"database":"ambari","table":"alert_current","type":"update","ts":1535363391,"xid":98489,"commit":true,"data":{"alert_id":122,"definition_id":82,"history_id":451,"maintenance_state":"ON","original_timestamp":1534996534493,"latest_timestamp":1535363390926,"latest_text":"Connection failed: [Errno 111] Connection refused to bdp03nn01:16000","occurrences":4991,"firmness":"HARD"},"old":{"latest_timestamp":1535363330967,"occurrences":4990}}
5.5 maxwell 的json数据说明
http://maxwells-daemon.io/dataformat/
5.6 参考
https://github.com/zendesk/maxwell
http://maxwells-daemon.io/compat/
https://github.com/zendesk/maxwell/releases
https://github.com/zendesk/maxwell/blob/master/CHANGELOG.md
http://maxwells-daemon.io/config/
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.3/bk_security/content/secure-kafka-produce-events.html
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_kafka-component-guide/content/ch_kafka-development.html