El servidor de Canal debe estar instalado antes de su uso, la última versión 1.1.4 de Canal está instalada y desplegada (1)
instalación
- La versión se puede ajustar según la situación. Para obtener la última versión, consulte: https://github.com/alibaba/canal/releases
#进入安装包
cd /opt
#下载安装包
wget https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.adapter-1.1.4.tar.gz
#创建安装目录
mkdir canal-adapter
#解压到指定目录
tar zxvf canal.adapter-1.1.5-SNAPSHOT.tar.gz -C ./canal-adapterConfiguración
Esencialmente un proyecto de arranque de primavera
vim conf/application.ymlserver:
port: 8081 #adapter 服务端口
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
logging:
level:
com.alibaba.otter.canal.client.adapter.es: DEBUG
canal.conf:
mode: tcp # tcp - canal server读取模式, kafka rocketMQ
flatMessage: true
zookeeperHosts: #对应集群模式下的zk地址, 如果配置了canalServerHost, 则以canalServerHost为准
syncBatchSize: 1 #批处理条数
retries: 0 #重试次数
timeout: #同步超时
accessKey:
secretKey:
consumerProperties: # 消息服务消费端地址，根据mode读取
# canal tcp consumer
canal.tcp.server.host: {canalServer}:11111 #cancal地址
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:
srcDataSources:
defaultDS: #支持多数据源
url: jdbc:mysql://{mysql}:3306/test?useUnicode=true
username: canal
password: canal
canalAdapters: # 同步适配器列表，同步到哪各数据源
- instance: example # canal 实例名或者 MQ topic 名
groups: # 适配器组，支持多个不同入库数据源
- groupId: g1 # 分组id, 如果是MQ模式将用到该值
outerAdapters: # 分组内适配器列表
- name: logger # 打印获取到的消息数据日志
# - name: rdb
# key: mysql1
# properties:
# jdbc.driverClassName: com.mysql.jdbc.Driver
# jdbc.url: jdbc:mysql://127.0.0.1:3306/mytest2?useUnicode=true
# jdbc.username: root
# jdbc.password: 121212
# - name: rdb
# key: oracle1
# properties:
# jdbc.driverClassName: oracle.jdbc.OracleDriver
# jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
# jdbc.username: mytest
# jdbc.password: m121212
# - name: rdb
# key: postgres1
# properties:
# jdbc.driverClassName: org.postgresql.Driver
# jdbc.url: jdbc:postgresql://localhost:5432/postgres
# jdbc.username: postgres
# jdbc.password: 121212
# threads: 1
# commitSize: 3000
# - name: hbase
# properties:
# hbase.zookeeper.quorum: 127.0.0.1
# hbase.zookeeper.property.clientPort: 2181
# zookeeper.znode.parent: /hbase
- name: es7 #用于匹配Adapter，如 ES7xAdapter
key: es-user #必须有key，否则是比es里面的配置
hosts: {es}:9200 # 127.0.0.1:9200 for rest mode
properties:
mode: rest # 9300对应transport or 9200对应rest
# security.auth: test:123456 # only used for rest mode
cluster.name: elasticsearch
# - name: kudu
# key: kudu
# properties:
# kudu.master.address: 127.0.0.1 # ',' split multi addressSegún el nombre del adaptador configurado, el adaptador cargará automáticamente todos los archivos de configuración que terminen en .yml en conf / {nombre}, como configuraciones relacionadas con conf / es.
Configuración del adaptador ES
Modificar la configuración de es
vim conf/es/mytest_user.ymldataSourceKey: defaultDS #源数据源的key, 对应上面配置的srcDataSources中的值
outerAdapterKey: es-user # 对应application.yml中es配置的key
destination: example # cannal的instance或者MQ的topic
groupId: g1 # 对应MQ模式下的groupId, 只会同步对应groupId的数据
esMapping:
_index: mytest_user # es 的索引名称
_type: _doc # es 的type名称, es7下无需配置此项
_id: id # es 的_id, 如果不配置该项必须配置下面的pk项_id则会由es自动分配
upsert: true # 支持不存在新增操作
# pk: id
sql: "select t.id,t.username,t.phone from t_user t" #需要更新的所有字段都需写上
# objFields:
# _labels: array:; # 数组或者对象属性, array:; 代表以;字段里面是以;分隔的
etlCondition: "where a.c_time>={}" # etl 的条件参数，通过服务的etl接口传入。
commitBatch: 3000 # 提交批大小La configuración de cada fuente de datos del adaptador producirá una identificación, y la regla de producción es destino + "-" + ID de grupo + "_" + base de datos + "-" + tabla.
- base de datos: el nombre de la biblioteca en defaultDS, como prueba en jdbc: mysql: // {mysql}: 3306 / test? useUnicode = true
- table: la tabla principal en sql, como t_user en select t.id como _id de t_user t.
puesta en marcha
bin/startup.shColección de problemas de la versión CanalAdapter1.1.5
éxito
tailf logs/adapter/adapter.log
2021-01-15 20:53:18.752 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":4,"username":"s44s08","sex":1,"phone":"138212828282","createtime":1610715198000}],"database":"test","destination":"example","es":1610715198000,"groupId":"g1","isDdl":false,"old":[{"username":"44s08","createtime":1610714369000}],"pkNames":["id"],"sql":"","table":"t_user","ts":1610715198636,"type":"UPDATE"}
2021-01-15 20:53:24.241 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.es.core.service.ESSyncService - DML: {"data":[{"id":4,"username":"s44s08","sex":1,"phone":"138212828282","createtime":1610715198000}],"database":"test","destination":"example","es":1610715198000,"groupId":"g1","isDdl":false,"old":[{"username":"44s08","createtime":1610714369000}],"pkNames":["id"],"sql":"","table":"t_user","ts":1610715198636,"type":"UPDATE"}
Affected indexes: mytest_user