Canal related understanding

Reprinted: http://www.importnew.com/25189.html
Overview
canal is an open source project under Alibaba, developed in pure Java. Based on database incremental log analysis, it provides incremental data subscription & consumption. Currently, it mainly supports MySQL (also supports mariaDB).

Origin: In the early days, Alibaba's B2B company had a business requirement for synchronization across computer rooms due to the deployment of dual computer rooms in Hangzhou and the United States. However, the early database synchronization business was mainly based on the trigger method to obtain incremental changes. However, since 2010, Alibaba-based companies began to gradually try to analyze the logs based on the database to obtain incremental changes for synchronization. The business of mass subscription & consumption has since opened a new era.

Log-based incremental subscription & consumption supported services:

database mirroring
, database real-time backup
, multi-level index (sellers and buyers have their own sub-database indexes),
search build ,
business cache refresh,
price changes and other important business news
Working principle
MySQL master and backup replication implementation:



from the upper layer From the point of view, replication is divided into three steps:

the master records the changes in the binary log (these records are called binary log events, which can be viewed by show binlog events);
the slave copies the master's binary log events to its relay log;
the slave redoes events in the relay log, changing the data to reflect its own.
How canal works The


principle is relatively simple:

canal simulates the interaction protocol of mysql slave, pretends to be mysql slave, sends dump protocol to
mysql master, mysql master receives dump request, and starts to push binary log to slave (that is, canal)
canal parses binary log object (original is byte stream)
architecture Design
Personal understanding, data incremental subscription and consumption should have the following points: the

incremental subscription and consumption module should include binlog log capture, binlog log analysis, event distribution filtering (EventSink), storage (EventStore) and other main modules.
If you need to ensure that HA can use Zookeeper to save the state of each sub-module, so that the entire incremental subscription and consumption module can be stateless, of course, the state of the consumer (client) can also be stored in zk.
On the whole, centralized management and resource allocation are carried out through a Manager System.
You can refer to the following figure:



canal architecture design


Description :

server represents a canal running instance, which corresponds to a jvm
instance and corresponds to a data queue (1 server corresponds to 1..n instances)
instance module:

eventParser (data source access, simulation The slave protocol interacts with the master, protocol analysis)
eventSink (Parser and Store linker, data filtering, processing, and distribution)
eventStore (data storage)
metaManager (incremental subscription & consumption information manager)



The entire parser process of EventParser can be roughly divided into several parts:

Connection obtains the location of the last successful parsing (if it is started for the first time, it obtains the initially specified location or the binlog site of the current database)
Connection establishes a connection, and the BINLOG_DUMP command
Mysql starts The Binary Log received by the push Binary Log
is parsed by the Binlog parser, and some specific information is added. It is
passed to the EventSink module for data storage, which is a blocking operation until the storage is successful
. After the storage is successful, the Binary Log location is regularly recorded.
EventSink Design


Description :

Data filtering : Support wildcard filter mode, table name, field content, etc.
Data routing/distribution: solve 1:n (one parser corresponds to multiple stores mode)
data merge: solve n:1 (multiple parsers correspond to one store)
data Processing: perform additional processing before entering the store, such as join
1 data 1:n business:

in order to make reasonable use of database resources, common businesses are generally isolated according to schema, and then performed at the upper level of mysql or at the level of dao. A data source routing that shields the impact of the physical location of the database on development. Alibaba mainly uses cobar/tddl to solve the problem of data source routing. Therefore, in general, multiple schemas will be deployed on a database instance, and each schema will be concerned by one or more business parties.

2 Data n:1 business:

Similarly, when the data scale of a business reaches a certain level, it will inevitably involve horizontal splitting and vertical splitting. When these split data needs to be processed, it is necessary to link multiple stores for processing. The site will become multiple copies, and the progress of data consumption cannot be guaranteed as orderly as possible. Therefore, in certain business scenarios, the split incremental data needs to be merged, such as sorting and merging according to timestamp/global id.

EventStore design
currently implements Memory memory, local file storage, and persistence to zookeeper to ensure Data cluster sharing.
RingBuffer design of memory memory:



three cursors are defined

Put: the last write position of the Sink module for data storage
Get: the last extraction position obtained by the data subscription
Ack: the last consumption position of the successful data
consumption Based on the implementation of Disruptor's RingBuffer , Straighten the RingBuffer to see:



Implementation Description:

Put/Get/Ack cursor is used to increment, and
the get operation of the long-type storage buffer is used, and the remainder or the AND operation is used. (And operation: cusor & (size – 1) , the size needs to be an index of 2, which is more efficient)
Instance design


instance represents an actual running data queue, including EventPaser, EventSink, EventStore and other components.
CanalInstanceGenerator is abstracted, mainly considering the management method of configuration:

1. Manager mode: Connect with your own internal web console/manager system. (alibaba internal use method)

2. spring method: define based on spring xml + properties, build spring configuration.

spring/memory-instance.xml All components (parser, sink, store) choose the memory version mode, record the site The memory mode is selected, and after restarting, it will return to the original location for parsing. Features: Fastest, least dependent
spring/file-instance.xml All components (parser, sink, store) choose file-based persistence mode, note that HA mechanism is not supported. Support stand-alone persistence
spring/default-instance .xml All components (parser, sink, store) have selected persistence mode. At present, the main way of persistence is to write to zookeeper to ensure data cluster sharing. Support HA
spring/group-instance.xml is mainly for multi-database needs When merging, multiple physical instances can be merged into one logical instance to provide client access. Scenario: branch database business. For example, the product data is divided into 4 libraries, and each library will have an instance. If you do not need a group, when you want to consume data in business, you need to start 4 clients and link 4 instance instances respectively. After using group, it can be merged into a logical instance on the canal server. You only need to start one client and link the logical instance. The
server design


server represents a running instance of canal, which is specially abstracted for the convenience of componentized use. Two implementations of Embeded (embedded) / Netty (network access):

Embeded : It has relatively high requirements for latency and availability, and it can hold distributed related technologies (such as failover)
Netty : Based on netty, it encapsulates a layer of network protocols, and the canal server ensures its availability. The pull model adopted, Of course, the latency will be slightly discounted, but this also depends on the situation.
Incremental subscription/consumption design For the


specific protocol format, please refer to: CanalProtocol.proto
get/ack/rollback protocol introduction:

Message getWithoutAck(int batchSize), allowing to specify batchSize, you can get multiple items at a time, and the object returned each time is Message, The contents included are:
a. batch id uniquely identifies
the specific data object of b. entries, the corresponding data object format: EntryProtocol.proto
void rollback(long batchId), according to fate, roll back the last get request, and retrieve the data again . Submit based on the batchId obtained by get to avoid misoperation of
void ack (long batchId), as long as you think, confirm that the consumption has been successful, and notify the server to delete the data. Submit based on the batchId obtained by get to avoid misoperation
. The get/ack/rollback protocol of canal is different from the conventional jms protocol, allowing get/ack to be processed asynchronously. rollback, which is called streaming api in the project.
The benefits of streaming api design:
Asynchronous get/ack reduces network delay and operation cost caused by ack (99% of the states are in normal state, abnormal rollbacks are individual cases, and there is no need to sacrifice the entire performance for individual cases)
After get gets the data, When there is a bottleneck in business consumption or multi-process/multi-threaded consumption is required, the get data can be polled continuously, and tasks can be sent continuously to improve parallelization. (A case of the author in the actual business: business data consumption needs to cross Sino-US network, so an operation is basically more than 200ms, in order to reduce the delay, it is necessary to implement parallelization)
Streaming api design:



each get operation will generate a mark in the meta, and the mark will be incremented to ensure that the mark will remain unchanged during the operation. Uniqueness
Each get operation will continue to fetch the cursor recorded in the previous mark operation. If the mark does not exist, the last ack cursor will continue to fetch backwards
. When ack is performed, the ack needs to be performed numerically in the order of the mark. , can not jump ack. ack will delete the current mark mark, and update the corresponding mark position to last ack cusor
Once an abnormal situation occurs, the client can initiate a rollback situation and reset the position: delete all marks, clear the get request position, The next request will continue to fetch from the last ack cursor. The
data format
canal adopts protobuff:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Entry
    Header
        logfileName [binlog file name]
        logfileOffset [binlog position]
        executeTime [changed]
        schemaName
        tableName
        eventType [insert/update/delete type]
    entryType [transaction header BEGIN/transaction tail END /Data ROWDATA]
    storeValue [byte data, expandable, the corresponding type is RowChange]   
RowChange
    isDdl [whether it is a ddl change operation, such as create table/drop table]
    sql [specific ddl sql]
    rowDatas [specific insert/update/delete Change data, can be multiple, 1 binlog event event can correspond to multiple changes, such as batch processing]
        beforeColumns [Array of Column types]
        afterColumns [Array of Column types]     
Column
    index      
    sqlType [jdbc type]
    name [column name]
    isKey [Whether the primary key]
    updated [Whether there has been a change]
    isNull [Whether the value is null]
    value [Specific Content, note that it is text]
canal-message example:

such as a table in a database:

1
2
3
4
5
6
7
8
9
mysql> select * from person;
+----+------+---- --+------+
| id | name | age | sex |
+----+------+------+------+
| 1 | zzh | 10 | m |
|  3 | zzh3 |   12 | f    |
|  4 | zzh4 |    5 | m    |
+----+------+------+------+
3 rows in set (0.00 sec)
更新一条数据（update person set age=15 where id=4）：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
****************************************************
* Batch Id: [2] ,count : [3] , memsize : [165] , Time : 2016-09-07 15:54:18
* Start : [mysql-bin.000003:6354:1473234846000(2016-09-07 15:54:06)]
* End : [mysql-bin.000003:6550:1473234846000(2016-09-07 15:54:06)]
**************************************************** **

===============> binlog[mysql-bin.000003:6354] , executeTime : 1473234846000 , delay : 12225ms
BEGIN ----> Thread id: 67
--- -------------> binlog[mysql-bin.000003:6486] , name[canal_test,person] , eventType : UPDATE , executeTime : 1473234846000 , delay : 12225ms
id : 4 type=int( 11)
name : zzh4 type=varchar(100)
age : 15 type=int(11) update=true
sex : m type=char(1)
----------------
END - ---> transaction id: 308
================> binlog[mysql-bin.000003:6550] , executeTime : 1473234846000 , delay : 12240ms
HA mechanism design
canal HA score In two parts, canal server and canal client have corresponding ha implementations:

canal server: In order to reduce requests for mysql dump, only one instance on different servers is required to be running at the same time, and the others are in standby state.
canal client: In order to ensure orderliness, an instance can only be run by one instance at a time. The canal client performs get/ack/rollback operations, otherwise the client's reception cannot be guaranteed in order.
The control of the entire HA mechanism mainly depends on several features of zookeeper, watcher and EPHEMERAL nodes (bound with session life cycle), you can read my previous articles on zookeeper.

Canal Server: Rough



steps:

When the canal server wants to start a canal instance, it will first try to start a judgment with zookeeper (implementation: create an EPHEMERA node, whoever is successfully created will be allowed to start)
After the zookeeper node is successfully created, the corresponding canal server will be Start the corresponding canal instance. If the canal instance is not successfully created, it will be in the standby state.
Once zookeeper finds that the node created by canal server A disappears, it immediately informs other canal servers to perform the operation in step 1 again, and re-select a canal server to start the instance. .
Every time the canal client connects, it will first ask the zookeeper who has started the canal instance, and then establish a link with it. Once the link is unavailable, it will try to connect again.
The canal client method is similar to the canal server method, and it also uses The way zokeeper preempts EPHEMERAL nodes is controlled.
The HA configuration architecture diagram (example) is as follows:



Canal other connection methods
Canal has several connection methods:

1. Single connection



2. Two clients + two instances + 1 mysql

When mysql changes, both clients can obtain To change



3. One server + two instances + two mysql + two clients



4. Standby configuration of instance



Overall architecture
From the overall architecture, canal is of this architecture (canal does not include an operation and maintenance console web for docking, but it needs to be It must be managed by a Manager in a distributed environment):



an overall manager system corresponds to n Canal Servers (physically, one server), then a Canal Server corresponds to n Canal Instances (destinations). Generally speaking The above is a three-layer structure, and the second layer also requires the Manager to coordinate the operation and maintenance management.

So with the rise of Docker technology, can you try the following architecture?



Running an instance service in a docker is equivalent to omitting the concept of the server layer.
Configure an instance in the Manager System, and directly call a docker to publish the instance, which includes sending configuration information to the instance and starting the instance service.
During the running process of the instance, the information of binlog filename + binlog position is periodically refreshed to zk.
If an instance fails, the instance itself reports an error or zk perceives that the node has disappeared, then restart a docker service according to the corresponding information, such as the binlog filename+binlog position saved in the previous step. Of course, some retry mechanisms can be appropriately added here.
When updating, similar to AB test, first shut down a docker, and then start a new updated replacement, step by step.
When it comes to sub-table and sub-library, multiple physical tables correspond to one logical table, and the results can be
stored Modification : memory, files, zk, or
tools other than docker added to MQ for management. For example, kubernetes
can also further add HA functions. Two dockers correspond to one mysql, which is the master and backup of each other, similar to Canal's HA architecture. If the timeliness is not a strong sticker scenario, considering the cost, this function may not be used.
Summary
Here is a summary of some of Canal's points, for reference only:

Principle: simulate the interactive protocol of mysql slave, disguise itself as mysql slave, and send dump protocol to mysql master; mysql master receives the dump request and starts to push binary log to slave ( That is, canal); parsing the binary log object (original is byte stream)
repeated consumption problem: solve it on the consumer side.
Using open source open-replicator to parse binlog
canal needs to maintain EventStore, which can be accessed in Memory, File, zk
canal needs to maintain the state of the client, and an instance can only be consumed by one consumer at the same time
Data transmission format: protobuff
supports binlog format Type: statement, row, mixed. Multiple additional functions can only be used under row, such as otter
binlog position can be saved in memory, file, zk
instance startup method: rpc/http;
Embedded ACK mechanism
, no alarm, no monitoring, these two functions need to be connected to external systems
to facilitate rapid deployment.