安装部署canal

准备

对于自建 MySQL , 需要先开启 Binlog 写入功能，配置 binlog-format 为 ROW 模式，my.cnf 中配置如下

[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复

- 注意：针对阿里云 RDS for MySQL , 默认打开了 binlog , 并且账号默认具有 binlog dump 权限 , 不需要任何权限或者 binlog 设置,可以直接跳过这一步

授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant

CREATE USER canal IDENTIFIED BY 'Canal_666';  
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;

启动

下载 canal, 访问 release 页面 , 选择需要的包下载, 如以 1.1.4 版本为例

wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz

或上传 canal.deployer-1.1.4.tar.gz 压缩包至/mydata/canal 目录下

修改配置文件

/canal/conf/example/instance.properties

## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info，需要改成自己的数据库信息
canal.instance.master.address = 127.0.0.1:3306 
canal.instance.master.journal.name = 
canal.instance.master.position = 
canal.instance.master.timestamp = 
#canal.instance.standby.address = 
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 
#username/password，需要改成自己的数据库信息
canal.instance.dbUsername = canal  
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*

单topic多分区配置

扫描二维码关注公众号，回复： 13343474 查看本文章

# 消息队列对应topic名
canal.mq.topic=binlog_mall_1
# 发送到哪一个分区，由于下面用hash做分区，因此不设
#canal.mq.partition=0
# 根据正则表达式做动态topic，目前采用单topic，因此也不设
#canal.mq.dynamicTopic=mall\\..*
# 10个分区
canal.mq.partitionsNum=3
# 各个表的主键，依照主键来做hash分区
canal.mq.partitionHash=mall.address:address_id,mall.orders:order_id,mall.order_product:order_product_id,mall.product:product_id,mall.mall_category:category_id,mall.mall_comment:comment_id,mall.mall_goods_category:goods_category_id,mall.mall_goods_info:goods_id,mall.mall_goods_wish:id,mall.mall_new_tags_v2:tags_id,mall.mall_topic:topic_id,mall.mall_topic_goods:id,mall.mall_user_cart_info:id

多topic多分区配置

# 消息队列对应topic名
#canal.mq.topic=binlog_mall_1
# 发送到哪一个分区，由于下面用hash做分区，因此不设
#canal.mq.partition=0
# 这样会为每个表生成一个topic ,库名.表名 例如：canal_db.canal_table,canal_db.wk_record
canal.mq.dynamicTopic=.*\\..*
# 3个分区
canal.mq.partitionsNum=3
# 各个表的主键，依照主键来做hash分区
canal.mq.partitionHash=mall.address:address_id,mall.orders:order_id

代码：

spring
# kafka配置
  kafka:
    bootstrap-servers: 192.168.54.150:9092
    consumer:
       # 多个topic
      topic: canal_db.canal_table,canal_db.wk_record
      # 是否自动提交offset
      enable-auto-commit: false
      properties:
        group-id: canal_table
        pollTimeout: 3000
      client-id: canal_table_1

监听

@KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'.split(',')}",
            groupId = "${spring.kafka.consumer.properties.group-id}",
            containerFactory = "batchFactory")
    public void listen(List<ConsumerRecord<String, Message>> records, Acknowledgment ack) {
        List<KafkaMessage> messages = transformMessage(records);
        try {
            if (!CollectionUtils.isEmpty(messages)) {
                for (KafkaMessage message : messages) {
                    long batchId = message.getId();
                    int size = message.getEntries().size();
                    if (batchId == -1 || size == 0) {

                        continue;
                    }
//                CanalKafkaUtils.printEntry(message.getEntries());
                    canalDealService.printEntries(message);
                }
            }
        } catch (Exception e) {
            log.error("kafka消息消费失败", e);
        } finally {
            //直接提交offset  https://blog.csdn.net/qq330983778/article/details/105937689/
            ack.acknowledge();
        }


    }

启动

sh bin/startup.sh

查看 server 日志

vi logs/canal/canal.log</pre>

2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

查看 instance 的日志

vi logs/example/example.log

2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....

关闭

sh bin/stop.sh

重点过滤指定库表

vim /canal/conf/example/instance.properties

canal.instance.filter.regex=canal_db.canal_table

vim /canal/conf/canal.properties

canal.instance.filter.query.dml=true

猜你喜欢