Canal介绍
QuickStart
几点说明：(mysql初始化)
a. canal的原理是基于mysql binlog技术，所以这里一定需要开启mysql的binlog写入功能，并且配置binlog模式为row.
[mysqld] log-bin=mysql-bin #添加这一行就ok binlog-format=ROW #选择row模式 server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复b. canal的原理是模拟自己为mysql slave，所以这里一定需要做为mysql slave的相关权限.
CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; FLUSH PRIVILEGES;
针对已有的账户可通过grants查询权限：
show grants for 'canal'
启动步骤：
1. 下载canal
下载部署包
wget http://canal4mysql.googlecode.com/files/canal.deployer-1.0.0.tar.gz
or
自己编译
git clone [email protected]:otter-projects/canal.git cd canal; mvn clean install -Dmaven.test.skip -Denv=release
编译完成后，会在根目录下产生target/canal.deployer-$version.tar.gz
2. 解压缩
mkdir /tmp/canal tar zxvf canal.deployer-1.0.0.tar.gz -C /tmp/canal
解压完成后，进入/tmp/canal目录，可以看到如下结构：
drwxr-xr-x 2 jianghang jianghang 136 2013-02-05 21:51 bin drwxr-xr-x 4 jianghang jianghang 160 2013-02-05 21:51 conf drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib drwxr-xr-x 2 jianghang jianghang 48 2013-02-05 21:29 logs
3. 配置修改
公用参数：
vi conf/canal.properties
################################################# ######### common argument ############# ################################################# canal.id= 1 canal.address= canal.port= 11111 canal.zkServers= # flush data to zk canal.zookeeper.flush.period = 1000 ## memory store RingBuffer size, should be Math.pow(2,n) canal.instance.memory.buffer.size = 32768 ## detecing config canal.instance.detecting.enable = false canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now() canal.instance.detecting.interval.time = 3 canal.instance.detecting.retry.threshold = 3 canal.instance.detecting.heartbeatHaEnable = false # support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery canal.instance.transactionn.size = 1024 # network config canal.instance.network.receiveBufferSize = 16384 canal.instance.network.sendBufferSize = 16384 canal.instance.network.soTimeout = 30 ################################################# ######### destinations ############# ################################################# canal.destinations= example canal.instance.global.mode = spring canal.instance.global.lazy = true ##修改为false，代表立马启动 #canal.instance.global.manager.address = 127.0.0.1:1099 canal.instance.global.spring.xml = classpath:spring/memory-instance.xml #canal.instance.global.spring.xml = classpath:spring/default-instance.xml
应用参数：
vi conf/example/instance.properties
################################################# ## mysql serverId canal.instance.mysql.slaveId = 1234 # position info canal.instance.master.address = 127.0.0.1:3306 #改成自己的数据库地址 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = # username/password canal.instance.dbUsername = retl #改成自己的数据库信息 canal.instance.dbPassword = retl #改成自己的数据库信息 canal.instance.defaultDatabaseName = #改成自己的数据库信息 canal.instance.connectionCharsetNumber = 33 #改成自己的数据库信息 canal.instance.connectionCharset = UTF-8 #改成自己的数据库信息 # table regex canal.instance.filter.regex = .*\\..* #################################################
说明：
- canal.instance.connectionCharset 代表数据库的编码方式对应到java中的编码类型，比如UTF-8，GBK , ISO-8859-1
- canal.instance.connectionCharsetNumber 代表数据库的编码方式对应mysql中的唯一id，详细的映射关系可查看：com.mysql.jdbc.CharsetMapping.INDEX_TO_CHARSET
针对常见的编码：
utf-8 <=> 33
gb2312 <=> 24
gbk <=> 28
4. 准备启动
sh bin/startup.sh
5. 查看日志
vi logs/canal/canal.log
2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server. 2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111] 2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
具体instance的日志：
vi logs/example/example.log
2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties] 2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties] 2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
6. 关闭
sh bin/stop.sh
几点说明：(mysql初始化)
a. canal的原理是基于mysql binlog技术，所以这里一定需要开启mysql的binlog写入功能，并且配置binlog模式为row.
[mysqld] log-bin=mysql-bin #添加这一行就ok binlog-format=ROW #选择row模式 server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复b. canal的原理是模拟自己为mysql slave，所以这里一定需要做为mysql slave的相关权限.
CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; FLUSH PRIVILEGES;
针对已有的账户可通过grants查询权限：
show grants for 'canal'
it's over.