canal-adapter1.1.4同步mariaDB10.0.25 到elasticsearch7.9.3

其他 2021-01-24 10:23:46 阅读次数: 0

使用canal-adapter1.1.4同步mariaDB10.0.25 到elasticsearch7.9.3


canal 工作原理

  1. canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议
  2. MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
  3. canal 解析 binary log 对象(原始为 byte 流)
    主要步骤: canal-server读取binlog —> canal-adapter读取canal-server 处理数据到es

相关版本

  1. mariaDB10.0.25
  2. canal-server1.1.4
  3. canal-adapter1.1.4
  4. elasticsearch7.9.3

前提条件

mysql配置 binlog_format 必须为ROW

[mysqld]
log_bin=mysql-bin  #[必须]启用二进制日志
server-id=222      #[必须]服务器唯一ID 2^32，默认是1，一般取IP最后一段
binlog_format=row #STATEMENT只包含修改信息 ROW 任何情况都可以被复制  MIXED 前两者的混合使用

创建测试数据库和表 后面都是基于这个表测试

-- 创建数据库
create database `log` DEFAULT CHARACTER SET utf8;
-- 进入数据库
use log;
-- 创建表
CREATE TABLE `test` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT NULL,
  `age` int(11) DEFAULT NULL,
  `update_time` datetime DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=367 DEFAULT CHARSET=latin1;

索引

PUT /canal-test
{
  "mappings": {
        "properties": {
        "name":{
          "type": "text"
        },
        "age":{
          "type": "integer"
        },
        "updateTime":{
          "type": "date"
        }
    }
  }
}

canal-server

启动canal-server

docker-compose方式启动

version: '3'
services:
  canal-server:
    image: canal/canal-server:v1.1.4
    container_name: canal-server
    ports:
      - 11111:11111
    environment:
      - canal.instance.mysql.slaveId=12
      - canal.auto.scan=false
      - canal.destinations=test
      - canal.instance.master.address=192.168.0.201:13306
      - canal.instance.dbUsername=root
      - canal.instance.dbPassword=root
      - canal.instance.connectionCharset=UTF-8
      - canal.mq.topic=test
      #- canal.instance.filter.regex=esen_approval.apt_approval
    volumes:
      - /data/local/canal-server/11111/conf/:/admin/canal-server/conf/
      - /data/local/canal-server/11111/logs/:/admin/canal-server/logs/

canal-client 测试代码

pom.xml

        <dependency>
            <groupId>com.alibaba.otter</groupId>
            <artifactId>canal.client</artifactId>
            <version>1.1.4</version>
        </dependency>

SimpleClient.java

package com.example.canalclienttest;

/**
 * 功能描述：
 *
 * @author Administrator
 * @date 2020/12/19
 */
import java.net.InetSocketAddress;
import java.util.List;
import java.util.stream.Collectors;

import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import com.alibaba.otter.canal.protocol.Message;

public class SimpleClient {
    
    
    static int count=1;
    public static void main(String[] args) throws Exception {
    
    
        CanalConnector connector = CanalConnectors.newSingleConnector(
                new InetSocketAddress("192.168.0.202", 11111), "test", "", "");

        connector.connect();
        connector.subscribe(".*\\..*");
        connector.rollback();

        while (true) {
    
    
            Message message = connector.getWithoutAck(100);
            long batchId = message.getId();
            if (batchId == -1 || message.getEntries().isEmpty()) {
    
    
                Thread.sleep(100);
                continue;
            }
            System.out.println(count++);
            printEntries(message.getEntries());
            connector.ack(batchId);
        }
    }

    private static void printEntries(List<Entry> entries) throws Exception {
    
    
        for (Entry entry : entries) {
    
    
            if (entry.getEntryType() != EntryType.ROWDATA) {
    
    
                continue;
            }

            RowChange rowChange = RowChange.parseFrom(entry.getStoreValue());
            for (RowData rowData : rowChange.getRowDatasList()) {
    
    
                switch (rowChange.getEventType()) {
    
    
                    case INSERT:
                    case UPDATE:
                        System.out.print("UPSERT ");
                        printColumns(rowData.getAfterColumnsList());

                        if ("retl_buffer".equals(entry.getHeader().getTableName())) {
    
    
                            String tableName = rowData.getAfterColumns(1).getValue();
                            String pkValue = rowData.getAfterColumns(2).getValue();
                            System.out.println("SELECT * FROM " + tableName + " WHERE id = " + pkValue);
                        }
                        break;

                    case DELETE:
                        System.out.print("DELETE ");
                        printColumns(rowData.getBeforeColumnsList());
                        break;

                    default:
                        break;
                }
            }
        }
    }

    private static void printColumns(List<Column> columns) {
    
    
        String line = columns.stream()
                .map(column -> column.getName() + "=" + column.getValue())
                .collect(Collectors.joining(","));
        System.out.println(line);
    }

}

canal-adapter

安装canal-adapter

百度云canal.adapter-1.1.4.tar.gz，提取码：i3be
解压

mkdir /usr/local/canal.adapter-1.1.4
tar zxvf canal.adapter-1.1.4.tar.gz  -C /usr/local/canal.adapter-1.1.4

得到目录
在这里插入图片描述

重新编译canal-adapter es源码 复制到plugin目录

  1. 下载canal1.1.4源码
git clone [email protected]:alibaba/canal.git
  1. 修改canal-canal-1.1.4\client-adapter\elasticsearch\pom.xml
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>7.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>7.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.9.3</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.9.3</version>
        </dependency>
  1. 修改canal-canal-1.1.4\client-adapter\elasticsearch\src\main\resources\META-INF\canal\com.alibaba.otter.canal.client.adapter.OuterAdapter
es7.9.3=com.alibaba.otter.canal.client.adapter.es.ESAdapter
  1. mvn 打包
    在canal的根目录运行
cd mvn clean package -pl com.alibaba.otter:client-adapter.elasticsearch -am
旧方法和新方法参考 改一下再打包
6.4.3
import org.elasticsearch.cluster.metadata.MappingMetaData;
restHighLevelClient.bulk(bulkRequest);
7.9.3
import org.elasticsearch.cluster.metadata.MappingMetadata;
restHighLevelClient.bulk(bulkRequest,RequestOptions.DEFAULT);
  1. 把打包的client-adapter.elasticsearch-1.1.4-jar-with-dependencies.jar复制到plugin目录
    注意:重名可以随便给jar取个名字
    例如client-adapter.elasticsearch-1.1.4-jar-with-dependencies7.9.3.jar
    在这里插入图片描述

配置canal-adapter

参考文档canalSync-E
./conf/application.yml

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null
canal.conf:
  mode: tcp # kafka rocketMQ
  canalServerHost: 192.168.0.202:11111              #canal 地址和端口
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  srcDataSources:
    defaultDS:
      url: jdbc:mysql://192.168.0.201:13306/log?useUnicode=true        #数据库
      username: root
      password: root
  canalAdapters:
  - instance: test            # canal instance Name or mq topic name  对应canal创建的文件夹
    groups:
    - groupId: g1
      outerAdapters:
        - name: es7.9.3		#适配器名称
          hosts: 192.168.0.202:9200                     # es 集群地址, 逗号分隔
          properties:
            mode: rest # transport # or rest

./conf/es/test.yml

dataSourceKey: defaultDS        # 源数据源的key, 对应上面配置的srcDataSources中的值
#outerAdapterKey: exampleKey     # 对应application.yml中es配置的key 
destination: test            # cannal的instance或者MQ的topic
groupId: g1                       # 对应MQ模式下的groupId, 只会同步对应groupId的数据
esMapping:
  _index: canal-test
  _type: _doc
  _id: _id
  sql: "
  select a.id as _id,a.name as name,a.age as age,a.update_time as updateTime
  from test as a
  "
  commitBatch: 3000

启动canal-adapte

adapter会自动加载 conf/es 下的所有.yml结尾的配置文件

#第一个窗口启动canal-adapte
chmod +x ./bin/* && ./bin/restart.sh
#第二个窗口输出日志
tail -f ./logs/adapter/adapter.log

对数据表进行curd

测试语句

#
insert into test values(0,RAND(),FLOOR(RAND()*100),now())
update set name='update_namexxx' where id=
deleet from test where id=xxx

报错解决

日志报错Not found the mapping info of index

2020-12-22 16:49:02.519 [pool-3-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.IllegalArgumentException: Not found the mapping info of index: canal-test
java.lang.RuntimeException: java.lang.IllegalArgumentException: Not found the mapping info of index: canal-test
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:111)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:58)
	at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:169)
	at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:148)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:201)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Not found the mapping info of index: canal-test
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getEsType(ESTemplate.java:498)
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getValFromData(ESTemplate.java:345)
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getESDataFromDmlData(ESTemplate.java:376)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.singleTableSimpleFiledInsert(ESSyncService.java:434)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:134)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:94)
	... 11 common frames omitted

修改 com.alibaba.otter.canal.client.adapter.es.support.ESConnection

mappingMetaData = mappings.get(index).get(type);
150行改为
mappingMetaData = mappings.get(index).get(“properties”);

日志报错c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.NullPointerException

2020-12-22 17:06:27.092 [pool-3-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - java.lang.NullPointerException
java.lang.RuntimeException: java.lang.NullPointerException
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:111)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:58)
	at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:169)
	at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:148)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:201)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getEsType(ESTemplate.java:504)
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getValFromData(ESTemplate.java:345)
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.getESDataFromDmlData(ESTemplate.java:376)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.singleTableSimpleFiledInsert(ESSyncService.java:434)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.insert(ESSyncService.java:134)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.sync(ESSyncService.java:94)
	... 11 common frames omitted

修改com.alibaba.otter.canal.client.adapter.es.support.ESTemplate

Map<String, Object> esMapping = mappingMetaData.getSourceAsMap();
Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");
502和503行改成
Map<String, Object> esMapping = mappingMetaData.getSourceAsMap();
//Map<String, Object> esMapping = (Map<String, Object>) sourceMap.get("properties");

日志报错options cannot be null

2020-12-22 17:13:59.639 [pool-3-thread-1] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - options cannot be null
java.lang.NullPointerException: options cannot be null
	at java.util.Objects.requireNonNull(Objects.java:228)
	at org.elasticsearch.client.Request.setOptions(Request.java:129)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1610)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
	at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:533)
	at com.alibaba.otter.canal.client.adapter.es.support.ESConnection$ESBulkRequest.bulk(ESConnection.java:422)
	at com.alibaba.otter.canal.client.adapter.es.support.ESTemplate.commit(ESTemplate.java:198)
	at com.alibaba.otter.canal.client.adapter.es.service.ESSyncService.commit(ESSyncService.java:853)
	at com.alibaba.otter.canal.client.adapter.es.ESAdapter.sync(ESAdapter.java:151)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.batchSync(AbstractCanalAdapterWorker.java:201)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$1(AbstractCanalAdapterWorker.java:62)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.alibaba.otter.canal.adapter.launcher.loader.AbstractCanalAdapterWorker.lambda$null$2(AbstractCanalAdapterWorker.java:58)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2020-12-22 17:13:59.640 [Thread-4] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - Outer adapter sync failed!  Error sync but ACK!

修改com.alibaba.otter.canal.client.adapter.es.support.ESConnection

return restHighLevelClient.bulk(bulkRequest,null);
422行改为
return restHighLevelClient.bulk(bulkRequest,RequestOptions.DEFAULT);

再次重启测试,同时检查增删改查数据是否同步 问题已解决

2020-12-22 17:27:23.504 [pool-3-thread-1] INFO  c.a.otter.canal.client.adapter.es.support.ESTemplate - index=canal-test,type=_doc
2020-12-22 17:27:23.788 [pool-3-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {
    
    "data":[{
    
    "id":272,"name":"a","age":1,"update_time":1608600443000}],"database":"log","destination":"test","es":1608629243000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1608629243464,"type":"INSERT"} 
Affected indexes: canal-test 
2020-12-22 17:27:23.789 [pool-3-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {
    
    "data":[{
    
    "id":273,"name":"a","age":1,"update_time":1608600443000}],"database":"log","destination":"test","es":1608629243000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1608629243503,"type":"INSERT"} 
Affected indexes: canal-test 
2020-12-22 17:27:23.852 [pool-3-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {
    
    "data":[{
    
    "id":274,"name":"a","age":1,"update_time":1608600443000}],"database":"log","destination":"test","es":1608629243000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1608629243852,"type":"INSERT"} 
Affected indexes: canal-test 
2020-12-22 17:27:29.330 [pool-3-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {
    
    "data":[{
    
    "id":276,"name":"a","age":1,"update_time":1608600449000}],"database":"log","destination":"test","es":1608629249000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1608629249328,"type":"INSERT"} 
Affected indexes: canal-test 
2020-12-22 17:27:29.332 [pool-3-thread-1] DEBUG c.a.otter.canal.client.adapter.es.service.ESSyncService - DML: {
    
    "data":[{
    
    "id":277,"name":"a","age":1,"update_time":1608600449000}],"database":"log","destination":"test","es":1608629249000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"test","ts":1608629249329,"type":"INSERT"} 
Affected indexes: canal-test

相关资源下载

canal.adapter-1.1.4.tar.gz，提取码：i3be
client-adapter.elasticsearch-1.1.4-jar-with-dependencies7.9.3.jar，提取码：3wa8

猜你喜欢

转载自blog.csdn.net/chen_cxl/article/details/111551409
今日推荐
openKylin 社区生态委员会第六次会议圆满召开
阿里云正式发布通义千问 2.5
Python 3.13 发布首个 Beta：实验性自由线程模式和 JIT、改进交互式解释器
Stack Overflow 拿我的代码去训练 AI 大模型，还封了我的账号
Pop!_OS 的 COSMIC 桌面完成 App Store 上架工作
报告：Django 仍然是 74% 开发者的首选
《2024 年一季度互联网投融资运行情况》研究报告
15 年前上了“FFmpeg 耻辱柱”，今天他还得谢谢咱——腾讯QQPlayer一雪前耻？
TIOBE 5 月榜单：Fortran “复活”进入 Top 10
GCC 14.1 发布
面壁智能发布 Eurux-8x22B 开源大模型 —— 堪称「理科状元」
开源日报 | 谷歌扶持鸿蒙上位；开源Rabbit R1；Docker加持的安卓手机；微软的焦虑和野心；海尔电器把开放平台关了
周排行
计算机组成与设计（七）—— 除法器
Integer Approximation(分治+枚举)
大话数据库索引
windows10系统JDK的配置及下载地址
mysql实现秒值转换中原六仔平台搭建
Codeforces Round #556 (Div. 1)
百练1064 网线主管
Codeforces 995F Cowmpany Cowmpensation
子集生成之增量构造法，位向量法，二进制法
ERROR: cmd.exe failed with args /c "/APK\gradle\rungradle.bat...
每日归档
更多
2024-05-10(38)
2024-05-09(35)
2024-05-08(42)
2024-05-07(14)
2024-05-06(40)
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022