准备
- 对于自建 MySQL , 需要先开启 Binlog 写入功能,配置 binlog-format 为 ROW 模式,my.cnf 中配置如下
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义,不要和 canal 的 slaveId 重复
-
- 注意:针对阿里云 RDS for MySQL , 默认打开了 binlog , 并且账号默认具有 binlog dump 权限 , 不需要任何权限或者 binlog 设置,可以直接跳过这一步
- 授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant
CREATE USER canal IDENTIFIED BY 'Canal_666';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
启动
- 下载 canal, 访问 release 页面 , 选择需要的包下载, 如以 1.1.4 版本为例
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz
或上传 canal.deployer-1.1.4.tar.gz 压缩包 至/mydata/canal 目录下
修改配置文件
/canal/conf/example/instance.properties
## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info,需要改成自己的数据库信息
canal.instance.master.address = 127.0.0.1:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#username/password,需要改成自己的数据库信息
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*
单topic多分区配置
# 消息队列对应topic名
canal.mq.topic=binlog_mall_1
# 发送到哪一个分区,由于下面用hash做分区,因此不设
#canal.mq.partition=0
# 根据正则表达式做动态topic,目前采用单topic,因此也不设
#canal.mq.dynamicTopic=mall\\..*
# 10个分区
canal.mq.partitionsNum=3
# 各个表的主键,依照主键来做hash分区
canal.mq.partitionHash=mall.address:address_id,mall.orders:order_id,mall.order_product:order_product_id,mall.product:product_id,mall.mall_category:category_id,mall.mall_comment:comment_id,mall.mall_goods_category:goods_category_id,mall.mall_goods_info:goods_id,mall.mall_goods_wish:id,mall.mall_new_tags_v2:tags_id,mall.mall_topic:topic_id,mall.mall_topic_goods:id,mall.mall_user_cart_info:id
多topic多分区配置
# 消息队列对应topic名
#canal.mq.topic=binlog_mall_1
# 发送到哪一个分区,由于下面用hash做分区,因此不设
#canal.mq.partition=0
# 这样会为每个表生成一个topic ,库名.表名 例如:canal_db.canal_table,canal_db.wk_record
canal.mq.dynamicTopic=.*\\..*
# 3个分区
canal.mq.partitionsNum=3
# 各个表的主键,依照主键来做hash分区
canal.mq.partitionHash=mall.address:address_id,mall.orders:order_id
代码:
spring
# kafka配置
kafka:
bootstrap-servers: 192.168.54.150:9092
consumer:
# 多个topic
topic: canal_db.canal_table,canal_db.wk_record
# 是否自动提交offset
enable-auto-commit: false
properties:
group-id: canal_table
pollTimeout: 3000
client-id: canal_table_1
监听
@KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'.split(',')}",
groupId = "${spring.kafka.consumer.properties.group-id}",
containerFactory = "batchFactory")
public void listen(List<ConsumerRecord<String, Message>> records, Acknowledgment ack) {
List<KafkaMessage> messages = transformMessage(records);
try {
if (!CollectionUtils.isEmpty(messages)) {
for (KafkaMessage message : messages) {
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
continue;
}
// CanalKafkaUtils.printEntry(message.getEntries());
canalDealService.printEntries(message);
}
}
} catch (Exception e) {
log.error("kafka消息消费失败", e);
} finally {
//直接提交offset https://blog.csdn.net/qq330983778/article/details/105937689/
ack.acknowledge();
}
}
- 启动
sh bin/startup.sh
- 查看 server 日志
vi logs/canal/canal.log</pre>
2013-02-05 22:45:27.967 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
- 查看 instance 的日志
vi logs/example/example.log
2013-02-05 22:50:45.636 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2013-02-05 22:50:45.810 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
关闭
sh bin/stop.sh
重点 过滤指定库表
vim /canal/conf/example/instance.properties
canal.instance.filter.regex=canal_db.canal_table
vim /canal/conf/canal.properties
canal.instance.filter.query.dml=true