Database series-use Canal Admin to achieve MySQL incremental data synchronization operation and maintenance management

Based on the previous article: https://blog.csdn.net/Coder_Boy_/article/details/111055381

Here is mainly based on Canal Admin to build a MySQL incremental data management operation and maintenance web platform

Canal Admin provides canal with operation-oriented functions such as overall configuration management, node operation and maintenance, and provides a relatively friendly WebUI operation interface, which is convenient for more users to operate quickly and safely

There will be some pits in the actual installation and use, so as a way 踩坑笔记to introduce the local installation and use.

canal-admin Need to depend on canal, and canalversion>=1.1.4

specific process

1. Download canal-admin

https://hub.fastgit.org/alibaba/canal/releases/download/canal-1.1.4/canal.admin-1.1.4.tar.gz

2. Unzip

3. Initialize the local database,
connect to the local database, importconf/canal_manager.sql

mysql -uroot -p

# 导入初始化SQL
> source conf/canal_manager.sql

cd to the unzipped directory D:\DevResource\ELK\canal.admin-1.1.4 ( your own installation directory )

4. Modify the configuration file
conf/application.yml

server:
  port: 8089
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8

spring.datasource:
  address: 127.0.0.1:3306
  database: canal_manager
  username: root
  password: 123456 ## 你自己的数据库密码
  driver-class-name: com.mysql.cj.jdbc.Driver
  url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&useSSL=false&serverTimezone=UTC
  hikari:
    maximum-pool-size: 30
    minimum-idle: 1

canal:
  adminUser: admin
  adminPasswd: admin

5. Start
bin/startup.sh
Visit http://127.0.0.1:8089/ , the default user and password: admin/123456If the access fails, check logsthe log file below to locate the cause of the error

Startup log

Note here: your database version uses the driver package of MySQL5.1 by default. If yours is MySQL8, you need to replace it with the following package

启动后打开浏览器访问 http://localhost:8089/

canal-adminAt this time 集群管理, Server管理there Instance管理is no data in, and the following configuration is required.

6.canal configuration file modification

Go to Canal to configure

Modify canalthe configuration file
conf/canal_local.properties

# register ip
canal.register.ip =127.0.0.1

# canal admin config
canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9
# admin auto register
canal.admin.register.auto = true
canal.admin.register.cluster =

canal_local.propertiesStart with configurationcanal
startup.sh local

Visit the canal-admin side, create a corresponding Server andcanal_local.properties中配置对应

After confirming, you can see that a Server appears automatically, and you can configure, modify, start, and view the log on this page.

Next, configure

Go to Instance列表add Instance, choose 载入模版to modify

The meta.dat in the above figure stores metadata information in the stand-alone mode. If it is a cluster mode, zookeeper is required to coordinate services and manage metadata. This article is a stand-alone build

The corresponding examples above must be consistent (names can be arbitrary, but they must be consistent)

Run Instanceand Server, check l ogs, if there is no abnormal information, the configuration is successful

After everything is configured, when the service of the server where your canal server is located is started again, it will be executed with the configuration file configured in web-ui first.

Source code extension

Check the source code and found a problem, as described in the following screenshot:

Source code tracking:

The following is the code logic for refreshing: it is not clear why the official page refresh will restart all instances, even if you stopped before

I hope everyone pays attention to this problem, otherwise I doubt: I have stopped Instance obviously, why is it still in operation, just look at the source code.

Record this bug

After server1 is started, the instance mounted under it cannot be stopped and will be automatically refreshed and restarted. I don’t know what the purpose of the official website is (there are still special scenarios). The problem has been submitted to Ali Canal issue