mysql-replication reads binlog in real time

mysql-replication is a pure Python implementation of the MySQL replication protocol built on PyMYSQL. Events such as insert, update, delete data and raw SQL queries can be received.

effect:

MySQL to NoSQL Database Replication
MySQL to Search Engine Replication
Cache Invalidation
Auditing
Real-time Analysis of Changes in the Database

Use environment:

MySQL 5.5, 5.6 and 5.7
Python >= 2.7
Python 3.3, 3.4, 3.5 and 3.6 (3.2 is not supported)
PyPy (which is indeed faster than the standard Python interpreter)
This project is used for business-critical production in some medium-sized internet companies. But all use cases are not fully tested in the real world.

prerequisites:

MySQL server configuration: 
[mysqld]
server-id         = 1
log_bin             = /var/log/mysql/mysql-bin.log
expire_logs_days = 10
max_binlog_size  = 100M
binlog-format    = row #Very important if you want to receive write, update and delete row events

Document address:

https://github.com/noplay/python-mysql-replication

https://python-mysql-replication.readthedocs.io/en/latest/

Install:

pip install mysql-replication

Example of use:

#!/usr/bin/env python3
# _*_ coding:utf8 _*_
import json
import sys,os
import pymysqlreplication
from pymysqlreplication import BinLogStreamReader
from pymysqlreplication.row_event import (DeleteRowsEvent,UpdateRowsEvent,WriteRowsEvent,TableMapEvent)
MYSQL_SETTINGS = {"host": "192.168.1.238","port": 3306,"user": "root","passwd": "123456"}   ##配置链接地址
def main():
    stream = BinLogStreamReader(connection_settings=MYSQL_SETTINGS,server_id=5,blocking=False,only_schemas = ["masterdb"],freeze_schema=True) 
    for binlogevent in stream:
        try:
            if binlogevent.schema == "masterdb":
                if isinstance(binlogevent, WriteRowsEvent):
                    print("类型:insert;库表:`%s`.`%s`;%s"%(binlogevent.schema,binlogevent.table,binlogevent.rows[0]["values"]))
                elif isinstance(binlogevent, DeleteRowsEvent):
                    if binlogevent.primary_key : ##如果有主键
                        print("类型:delete;库表:`%s`.`%s`;主键:%s;%s"%(binlogevent.schema,binlogevent.table,binlogevent.primary_key,binlogevent.rows[0]["values"]))
                elif isinstance(binlogevent, UpdateRowsEvent):
                    if binlogevent.primary_key:
                        print("类型:update;库表:`%s`.`%s`;主键:%s;%s"%(binlogevent.schema,binlogevent.table,binlogevent.primary_key,binlogevent.rows[0]))
                elif isinstance(binlogevent,TableMapEvent):
                    pass
                else:
                    binlogevent.dump()
        except Exception:
            pass
    stream.close()
if __name__ == "__main__":
    main()

 

BinLogStreamReader parameter description:

ctl_connection_settings: connection settings for cluster save mode information 
resume_stream: start from latest event or old available event from location or binlog 
log_file: set replication start log file 
log_pos: set replication start log pos (resume_stream should be true) 
auto_position: use master_auto_position gtid setting position 
blocking: reading blocked on stream 
only_events: array of allowed events 
ignored_events: array of ignored events 
only_tables: array containing tables to watch (only for binlog_format ROW) 
ignored_tables: array containing tables to skip 
only_schemas : Array containing schemas to watch 
ignored_schemas : Array containing schemas to skip 
freeze_schema : If true, ALTER TABLE is not supported. faster. 
skip_to_timestamp: Skip all events until the specified timestamp is reached. 
report_slave: Report slaves in SHOW SLAVE HOSTS. 
slave_uuid: Report slave_uuid in SHOW SLAVE HOSTS. 
fail_on_table_metadata_unavailable: should throw an exception if we can't get table information about row_events
slave_heartbeat: (seconds) The master should actively send heartbeat connections. This also reduces GTID replication traffic when replication resumes (in the case where many events are skipped in the binlog). See MASTER_HEARTBEAT_PERIOD in the mysql documentation for semantics