DB canal project distributed cluster synchronization of data

canal incremental Alibaba binlog mysql database subscriptions and Consumer components.
GitHub: https://github.com/alibaba/canal
https://github.com/alibaba/canal/wiki/ Profile

working principle

mysql standby duplication implementations
from the top view, copy into three steps:

The change to the master record binary log (binary log) in (these records is called binary log events, binary log events, can be viewed by show binlog events);
The master slave copy of the binary log events to its relay log (relay log);
slave relay redo log events, will change to reflect its own data.

canal works: The
principle is relatively simple:

canal interactive simulation mysql slave protocol, disguised himself as mysql slave, to send dump to mysql master agreement
mysql master dump request is received, began to push binary log to the slave (ie canal)
Analytical canal binary log object (for the original byte stream)

use

Log on incremental subscription & support consumer business:

Database Mirroring
Database real-time backup
Multi-level index (sellers and buyers each sub-library index)
search build
Business cache refresh
Price changes and other important business news

Architecture

Description:

Representative canal running a server instance of a corresponding jvm
instance corresponding to a data queue (corresponding to a server instance number 1 ... n)

Module instance:

eventParase (source access data, and the analog master slave protocol interact protocol analysis)
eventSink (Parser and Store linker, data filtering, processing, distribution of the work)
eventStore (data storage)
metaManager (incremental subscription & Consumer Information Manager)

EventParser Design

The general process

The whole process can be divided parser steps:

Connection on obtaining a successful parse position (if the first time you start, you get binlog site of the initial specified location or the current database)
Connection建立链接，发送BINLOG_DUMP指令
// 0.write command number
// 1.write 4 bytes bin-log position to start at
// 2.write 2 bytes bin-log flags
// 3.write 4 bytes server id of the slave
// 4.write bin-log file name
Mysql began to push Binaly Log
Binaly Log received by Binlog parser parses protocol, add some specific information
@ supplemental field names, field types, primary key information, unsigned type of treatment
EventSink transmitted to the data storage module is a blocking operation, until the storage successful
After successfully stored, the timer record position Binaly Log

The Binlay Log mysql network protocols:

4byte header protocol is described mainly binlog entire length of the packet network.
binlog event structure, detailed information please refer to:
https://dev.mysql.com/doc/internals/en/binary-log.html https://dev.mysql.com/doc/internals/en/event-structure. HTML https://dev.mysql.com/doc/internals/en/binlog-event.html

EventSink Design

Description:

Data filtering: Support for wildcard filtering mode, indicates that field contents
Data routing / Distribution: Solution 1: n (1 th parser modes corresponding to the plurality of store)
Data Merge: Solution n: 1 (corresponding to a plurality of parser store)
Data processing: additional treatment prior to entering the store, such as the join

Data 1: n Business

For rational use of database resources, common services are in accordance with the general schema isolation, and then on the upper or mysql dao this level, be a data source routing, shielding effects of the physical location of database development, mainly through Ali cobar / tddl data source routing to resolve the problem.

So, generally on a database instance, will deploy multiple schema, each schema will be concerned by the one or more business parties

Data n: 1 business

Similarly, when the data size of a business reaches a certain magnitude, when bound to issues related to the horizontal and vertical split split, split for these data need to be addressed, we need to link multiple store for processing, consumption the site will become multiple copies, and data consumption of progress can not be as orderly guarantee.

Therefore, in certain service scenarios, you need to be split after the incremental data merge process, such as sorting according to the time stamp merging / global id.

EventStore Design

1. Currently only implements memory Memory mode, the follow-up plans to increase the local file storage, mixed Mixed Mode
1. It draws on the realization of ideas RingBuffer of the Disruptor

RingBuffer Design

It defines three cursor

Put: Sink module data writing position stored in the last
Get: subscription data acquired last extraction position
Ack: The last time consumer location data consumption of success

RingBuffer realization draw Disruptor will RingBuffer straightened look:

Satisfies Ack <= Get <= Put
Put - Ack <= RingBuffer Size

Implementation Notes

Put / Get / Ack cursor is used to increment, using long type memory
buffer the get operation, or by taking over the operation. (Operation: cusor & (size - 1), size needs to be a power of two, more efficient)

Instance Design

instance data representing the actual operation of a queue, including EventParser, EventSink, EventStore other components.
Abstract CanalInstanceGenerator, the main consideration is management configuration:

manager way: and your own internal docking web console / manager system
spring Method: spring xml + properties defined construct spring configuration

Server Design

Representative examples of a server running the canal, in order to facilitate use of the component, specifically the abstract Embeded (embedded) / Netty (Network Access) two implementations

Embeded: for latency and high availability are requirements, but also hold their own live distributed technologies (such as failover)
Netty: netty encapsulating layer based network protocol, a canal server guarantee its availability, pull model used, of course, will be a little bit of discount latency, but this is also subject to availability. (Ali and notify the Department of metaq, typical push / pull model, is also gradually move closer to the pull model, push there will be some problems when large volumes of data)

Subscribe to increment / consumer design

Agreement codec: protobuf

subscribe / unsubscrible only the first need
subscribe allow repeated calls to each submit a new filter

Specific protocol format, see: CanalProtocol.proto

get / ack / rollback protocol description:

Message getWithoutAck (int batchSize), allows you to specify batchSize, one can obtain a plurality of objects each returned to the Message, the content is included:
. BATCH ID A unique identification
. B entries particular data object, the data object corresponding to the format: EntryProtocol.proto
void rollback (long batchId), Gu Ming Incredibles, to roll back the last of get requests, retrieve the data. Based get acquired batchId submit to prevent misuse
void ack (long batchId), Gu Ming Incredibles, confirmed that consumption has been successful, the notification server delete data. Based get acquired batchId submit to prevent misuse

The canal get / ack / rollback conventional protocols and different protocols jms allow get / ack asynchronous processing, such as successive calls may get several subsequent asynchronous filed ack / rollback sequentially project called streaming api.

Streaming api design benefits:

get / ack Asynchronized, ack reduce the delay caused by the network and operating costs (99% of the states are in a normal state, abnormal belongs rollback individual cases is not necessary to sacrifice the overall performance of the individual case)
After obtaining the data, there is a bottleneck or when the consumer get the business needs multi-process / multi-thread consumption, you could go get polling data, sending back non-stop mission to improve the parallelism (in the case of a real business are: Chinese and American consumption needs service data across the network, so that more than one operation substantially 200ms, in order to reduce the delay, it is necessary to implement parallelization)

Streaming api design:

Each get operation will have a mark in the meta, mark mark is incremented ensure the uniqueness of the mark during operation
Of each get operation, the cursor will continue to take back the last operation record mark, if the mark is not present, then continue to take back the last ack cursor
When ACK, is required in order to mark ACK sequence number, you can not jump ack. Ack will delete the current mark tag, and updates the corresponding mark position last ack cusor
Once the abnormal situation, the client can initiate a rollback situation, re-set: delete all mark, get request to clean up the position, the next request will take back from last ack cursor

Data Object Format: EntryProtocol.proto

Entry
	Header
		logfileName [binlog文件名]
		logfileOffset [binlog position]
		executeTime [binlog里记录变更发生的时间戳]
		schemaName [数据库实例]
		tableName [表名]
		eventType [insert/update/delete类型]
	entryType 	[事务头BEGIN/事务尾END/数据ROWDATA]
	storeValue 	[byte数据,可展开，对应的类型为RowChange]
RowChange
isDdl		[是否是ddl变更操作，比如create table/drop table]
sql		[具体的ddl sql]
rowDatas	[具体insert/update/delete的变更数据，可为多条，1个binlog event事件可对应多条变更，比如批处理]
beforeColumns [Column类型的数组]
afterColumns [Column类型的数组]


Column
index		[column序号]
sqlType		[jdbc type]
name		[column name]
isKey		[是否为主键]
updated		[是否发生过变更]
isNull		[值是否为null]
value		[具体的内容，注意为文本]

Description:

Can provide a database field contents before the change and after the change, a completion for information name, isKey and other binlog not found
Changes can provide ddl statement

HA mechanism design

The canal ha divided into two parts, canal server and Client canal respectively corresponding to achieve ha

canal server: To reduce the mysql dump request for, instance on a different server at the same time requires only one is running, the other is in a standby state
canal client: In order to ensure orderly, one instance the same time can only get / ack / rollback operated by a canal client, or the client can not ensure the orderly reception.

HA control the entire mechanism is mainly dependent on several features of zookeeper, watcher and EPHEMERAL node (and session lifecycle bindings)

General steps:

canal server To start a canal instance are Xianxiang zookeeper once try to start the judgment (to achieve: create EPHEMERAL node, who create success on who is allowed to start)
After creating a successful zookeeper node, the corresponding canal server started corresponding canal instance, did not create a successful canal instance will be in a standby state
Once zookeeper found node canal server A create disappearance, immediately notify the other of the canal server operation Step 1 again, to re-elect a canal server start instance.
canal client each time connect, will first ask the zookeeper who is currently starting a canal instance, and then establish their links, once the link is unavailable, it will attempt to re-connect.

Canal Client manner and in a manner analogous canal server, also using the preemption EPHEMERAL node zookeeper controlled manner.