Alibaba Open Source Project: Incremental Subscription & Consumption Based on MySQL Database Binlog

Background

   In the early days, Alibaba's B2B company had business requirements for synchronization across computer rooms due to the deployment of dual computer rooms in Hangzhou and the United States. However, the early database synchronization business was mainly based on the trigger method to obtain incremental changes. However, since 2010, Alibaba-based companies began to gradually try to analyze the logs based on the database to obtain incremental changes for synchronization. The business of mass subscription & consumption has since opened a new era. ps. The synchronization currently used internally already supports log parsing of some versions of mysql5.x and oracle

. Incremental log subscription & consumption-based services:
database mirroring
, database real-time backup
, multi-level indexing (sellers and buyers have their own sub-database indexes)
search Build
business cache refresh Important business news such as
price changes
Project introduction

   name: canal [kə'næl]
   translation meaning: waterway/pipe/ditch
   Language: pure java development
   orientation: based on database incremental log analysis, providing incremental data subscription & consumption, At present, it mainly supports the

working principle of mysql and
the implementation of mysql master and backup replication.


From the upper layer, the replication is divided into three steps:
the master records the changes in the binary log (these records are called binary log events, binary log events, which can be displayed by show binlog events to view);
the slave copies the master's binary log events to its relay log;
The slave redoes events in the relay log, changing the data to reflect its own.
The working principle of canal: the

principle is relatively simple:
canal simulates the interactive protocol of mysql slave, pretends to be mysql slave, sends dump protocol to
mysql master, mysql master receives the dump request, and starts to push binary log to slave (that is, canal)
canal parsing binary log object (original is byte stream)
architecture


Description :

server represents a canal running instance, corresponding to a jvm
instance and corresponding to a data queue (1 server corresponds to 1..n instances)
instance module:
eventParser (data source access , simulate the interaction between slave protocol and master, protocol analysis)
eventSink (Parser and Store linker, perform data filtering, processing and distribution work)
eventStore (data storage)
metaManager (incremental subscription & consumption information manager)
knowledge popularization
mysql Introduction to Binlay Log
http://dev.mysql.com/doc/refman/5.5/en/binary-log.html
http://www.taobaodba.com/html/474_mysqls-binary-log_details.html
Simply put:
MySQL's binlog is multi-file storage. To locate a LogEvent, you need to use binlog filename + binlog position to locate
MySQL's binlog data format. According to the method of generation, it is mainly divided into: statement-based, row-based, and mixed.
Java code Favorite code
mysql> show variables like 'binlog_format'; 
+---------------+-------+ 
| Variable_name | Value | 
+------ ---------+-------+ 
| binlog_format | ROW | 
+---------------+-------+ 
1 row in set (0.00 sec) 
Currently canal can only support incremental subscription in row mode (statement only has sql, no data, so the original change log cannot be obtained)


EventParser design
Rough process:

The entire parser process can be roughly divided into several steps:
Connection acquisition The location where the last parsing succeeded (if it is started for the first time, it will obtain the initially specified location or the binlog site of the current database)
Connection establishes a link and sends the BINLOG_DUMP command
// 0. write command number
// 1. write 4 bytes bin-log position to start at
// 2. write 2 bytes bin-log flags
// 3. write 4 bytes server id of the slave
// ​​4. write bin-log file name
Mysql starts pushing The Binaly Log
received by Binaly Log is parsed by the Binlog parser, and some specific information is
added. After the storage is successful, regularly record the Binlay Log network protocol of Binaly Log location mysql: Description: The protocol 4byte header in the figure mainly describes the length binlog event structure of the entire binlog network packet. For details, please refer to: http://dev.mysql .com/doc/internals/en/binary-log.html EventSink Design Description : Data filtering: support wildcard filtering mode, table name, field content, etc. Data routing/distribution: solve 1:n (one parser corresponds to multiple stores mode) data merge: solve n:1 (multiple parsers correspond to 1 store)














Data processing: Additional processing is performed before entering the store, such as join
data 1:n business.
  In order to make reasonable use of database resources, common business is generally isolated according to schema, and then a mysql upper layer or dao layer is performed. Data source routing, shielding the impact of the physical location of the database on development, Alibaba mainly solves the problem of data source routing through cobar/tddl.
  Therefore, in general, multiple schemas will be deployed on a database instance, and each schema will be concerned by one or more business parties.

Data n:1 business
  Similarly, when the data scale of a business reaches a certain magnitude, it will inevitably be When it comes to horizontal splitting and vertical splitting, when the split data needs to be processed, it is necessary to link multiple stores for processing, the consumption sites will become multiple copies, and the progress of data consumption cannot be exhausted. Possibly ordered guarantees.
  Therefore, in certain business scenarios, the split incremental data needs to be merged, such as sorting and merging according to timestamp/global id.

EventStore Design
1. Currently only the Memory memory mode is implemented, and subsequent plans to increase local file storage , mixed mixed mode
2. Drawing on the implementation idea of ​​Disruptor's RingBuffer
RingBuffer design:

defines 3 cursors
Put : The last write position of the Sink module for data storage
Get : The last extraction position of the data subscription acquisition
Ack : The data consumption is successful The position of the last consumption of .
Drawing on the implementation of Disruptor's RingBuffer, straighten the RingBuffer to see:

Implementation description:
The Put/Get/Ack cursor is used to increment, using
the get operation of the long-type storage buffer, by taking the remainder or the AND operation. (And operation: cusor & (size - 1) , the size needs to be an index of 2, and the efficiency is relatively high)
Instance design



instance represents an actual running data queue, including EventPaser, EventSink, EventStore and other components.
CanalInstanceGenerator is abstracted, mainly considering the management mode of configuration:
manager mode: docking with your own internal web console/manager system. (Currently mainly used within the company)
spring method: define based on spring xml + properties, build spring configuration.
Server design


server represents a running instance of canal, in order to facilitate the use of components, Embeded (embedded) / Netty is specially abstracted Two implementations of (network access)
Embeded: It has relatively high requirements for latency and availability, and it can hold distributed related technologies (such as failover)
Netty: Based on netty, it encapsulates a layer of network protocol, which is guaranteed by canal server Its availability, the pull model used, and of course the latency will be slightly discounted, but this also depends on the situation. (Alibaba's notify and metaq, the typical push/pull model, are gradually moving closer to the pull model. Push will have some problems when the amount of data is large.)
Incremental subscription/consumption design

The specific protocol format can be found in : CanalProtocol.proto
The introduction of the get/ack/rollback protocol:
Message getWithoutAck(int batchSize), allows to specify batchSize, multiple items can be obtained at one time, each time the returned object is Message, which contains:
a. batch id uniquely identifies
b. specific data of entries Object, the corresponding data object format: EntryProtocol.proto
void rollback(long batchId), according to the fate, roll back the last get request and get the data again. Submit based on the batchId obtained by get to avoid misoperation of
void ack (long batchId), as long as you think, confirm that the consumption has been successful, and notify the server to delete the data. Submit based on the batchId obtained by get to avoid misoperation
. The get/ack/rollback protocol of canal is different from the conventional jms protocol, allowing get/ack to be processed asynchronously. Rollback is called streaming api in the project.
The benefits of streaming api design:
get/ack is asynchronous, reducing network delay and operation cost caused by ack (99% of the states are in normal state, abnormal rollback belongs to In individual cases, there is no need to sacrifice the entire performance for individual cases.)
After get data is acquired, there is a bottleneck in business consumption or when multi-process/multi-threaded consumption is required, the get data can be polled continuously, and tasks can be sent continuously to improve the Parallelization. (A case of the author in the actual business: business data consumption needs to cross the China-US network, so an operation is basically more than 200ms, in order to reduce the delay, it is necessary to implement parallelization)
Streaming API design:

Each get operation will generate a mark in the meta, and the mark will be incremented to ensure the uniqueness of the mark during the running process.
Each get operation will continue to fetch the cursor recorded in the previous mark operation, if the mark does not exist , then when the last ack cursor continues to fetch backwards
for ack, it is necessary to perform the ack in the order of the marks, and the ack cannot be skipped. The ack will delete the current mark and update the corresponding mark position to the last ack cusor
once an exception occurs In this case, the client can initiate a rollback situation and reset it: delete all marks, clear the get request position, and the next request will continue to fetch from the last ack cursor.
Data object format: EntryProtocol.proto
Java code Collection code
Entry 
    Header 
        logfileName [ binlog file name] 
        logfileOffset [binlog position] 
        executeTime [changes occurred] 
        schemaName  
        tableName 
        eventType [insert/update/delete type] 
    entryType [transaction header BEGIN/transaction tail END/data ROWDATA] 
    storeValue [byte data, expandable, corresponding type for RowChange] 
     
RowChange 
    isDdl [Whether it is a ddl change operation, such as create table/drop table] 
    sql [specific ddl sql] 
    rowDatas [specific insert/update/delete change data, which can be multiple, and one binlog event event can correspond to multiple changes , such as batch processing] 
        beforeColumns [Array of Column type] 
        afterColumns [Array of Column type] 
         
Column  
    index        
    sqlType [jdbc type] 
    name [column name] 
    isKey [Whether the primary key] 
    updated [Whether there has been a change] 
    isNull [Whether the value is null ] 
    value [specific content, note that it is text] 
Description:
It can provide the field content before and after the database change, and complete the name, isKey and other information that is not in the binlog
Can provide the ddl change statement

HA mechanism
to design canal ha Divided into two parts, canal server and canal client have corresponding ha implementations respectively
canal server: In order to reduce requests for mysql dump, only one instance on different servers is required to be running at the same time, and the others are in standby state.
canal client: In order to ensure orderliness, an instance can only be run by one instance at a time. The canal client performs get/ack/rollback operations, otherwise the client's reception cannot be guaranteed in order.
The control of the entire HA mechanism mainly depends on several features of zookeeper, watcher and EPHEMERAL nodes (bound with session life cycle), you can read my previous articles on zookeeper.

Canal Server: Rough

steps:
When the canal server wants to start a canal instance, it first tries to start a judgment with zookeeper (implementation: create an EPHEMERAL node, whoever is successfully created will be allowed to start)
After the zookeeper node is successfully created, the corresponding canal server will be Start the corresponding canal instance. If the canal instance is not successfully created, it will be in the standby state.
Once zookeeper finds that the node created by canal server A disappears, it immediately informs other canal servers to perform the operation in step 1 again, and re-select a canal server to start the instance. .
Every time the canal client connects, it will first ask the zookeeper who has started the canal instance, and then establish a link with it. Once the link is unavailable, it will try to connect again.
The canal client method is similar to the canal server method, and it also uses The way zokeeper preempts EPHEMERAL nodes is controlled.

The code of the final
project : https://github.com/alibabatech/canal
Here is an example of how to quickly start Canal Server and Canal Client. If you have any questions, please feel free to contact
Quick Start
http://agapple.iteye.com/blog /1796070
https://github.com/alibabatech/canal/wiki/QuickStart
Client Example
http://agapple.iteye.com/blog/1796620
https://github.com/alibabatech/canal/wiki/ClientExample