Data synchronization tool: Canal installation

Hello everyone , and welcome to this blog. The blogger is a novice who has just entered the big data industry. He uses his free time to share what he has learned and helps students who are just like the blogger who is just in the beginning stage. The level is not high. , If there are any mistakes and omissions, please feel free to enlighten me. At present, the personal blog only has CSDN:, https://zhenyu.blog.csdn.net/thank you for your support, thank you, and
send you a sentence: today’s matter, today

This blog mainly explains: Data synchronization tool: Canal installation

aims:

• Install Canal

Important version update instructions

------ 1.Canal 1.1.x version ( release_note ), there is a big breakthrough in performance and function. Important improvements include:

• The overall performance test & optimization has improved by 150%. #726 Reference: Performance
• Native support for prometheus monitoring #765 Prometheus QuickStart
• Native support for kafka message delivery #695 Canal Kafka/RocketMQ QuickStart
• Native support for binlog subscription of aliyun rds (solve automatic active/standby switching/oss binlog offline analysis) Reference: Aliyun RDS QuickStart
• Native support for docker image#801 Reference: Docker QuickStart

2.canal 1.1.4 version, ushered in the most important WebUI capabilities, introduced the canal-admin project, supports WebUI-oriented canal dynamic management capabilities, and supports online white-screen operation and maintenance capabilities such as configuration, tasks, logs, etc. Specific document: Canal Admin Guide

Note: The version used in this study canal1.0.24

Environmental requirements

• Download: click to download Cancal_ extraction code: yujv
• Install ZooKeeper
• unzip

mkdir /export/servers/canal
tar -zxvf canal.deployer-1.0.24.tar.gz  -C /export/servers/canal/

• After decompression is complete, enter the /export/servers/canal/ directory, you can see the following structure

[root@node1 canal]# tree conf/ 
conf/
├── canal.properties
├── example
│   └── instance.properties
├── logback.xml
└── spring
    ├── default-instance.xml
    ├── file-instance.xml
    ├── group-instance.xml
    ├── local-instance.xml
    └── memory-instance.xml

• First look canal.propertiesat the first four configuration items of the common attribute:

canal.id= 1
canal.ip=
canal.port= 11111
canal.zkServers=

canal.id is the number of canal. In a cluster environment, the id of different canal is different. Note that it is different from the server_id of MySQL.
Ip is not specified here, and the default is this machine, for example, the above is 192.168.1.120, and the port number is 11111. zk is used for canal cluster.

• Let's look at the configuration related canal.propertiesto destinations :

#################################################
#########       destinations        ############# 
#################################################
canal.destinations = example
canal.conf.dir = ../conf
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.global.mode = spring 
canal.instance.global.lazy = false
canal.instance.global.spring.xml = classpath:spring/file-instance.xml

------ Here canal.destinations= examplecan set multiple, such as example1, example2, you need to create two corresponding folders, and each folder has a instance.propertiesfile.

• The global canal instance management uses spring, here will file-instance.xmleventually instantiate all the destinations instances:

<!-- properties -->
<bean class="com.alibaba.otter.canal.instance.spring.support.PropertyPlaceholderConfigurer" lazy-init="false">
	<property name="ignoreResourceNotFound" value="true" />
    <property name="systemPropertiesModeName" value="SYSTEM_PROPERTIES_MODE_OVERRIDE"/><!-- 允许system覆盖 -->
    <property name="locationNames">
    	<list>
        	<value>classpath:canal.properties</value>                     <value>classpath:${canal.instance.destination:}/instance.properties</value>
         </list>
    </property>
</bean>

<bean id="socketAddressEditor" class="com.alibaba.otter.canal.instance.spring.support.SocketAddressEditor" />
<bean class="org.springframework.beans.factory.config.CustomEditorConfigurer"> 
   <property name="propertyEditorRegistrars">
	   <list>
    		<ref bean="socketAddressEditor" />
       </list>
   </property>
</bean>
<bean id="instance" class="com.alibaba.otter.canal.instance.spring.CanalInstanceWithSpring">
	<property name="destination" value="${canal.instance.destination}" />
    <property name="eventParser">
    	<ref local="eventParser" />
    </property>
    <property name="eventSink">
        <ref local="eventSink" />
    </property>
    <property name="eventStore">
        <ref local="eventStore" />
    </property>
    <property name="metaManager">
        <ref local="metaManager" />
    </property>
    <property name="alarmHandler">
        <ref local="alarmHandler" />
    </property>
</bean>

------For example, if canal.instance.destinationequal to example, the example/instance.propertiesconfiguration file will be loaded

• Modify the instance configuration file
  vi conf/example/instance.properties

## mysql serverId，这里的slaveId不能和myql集群中已有的server_id一样
canal.instance.mysql.slaveId = 1234

#  按需修改成自己的数据库信息
#################################################
...
canal.instance.master.address=192.168.1.120:3306
# username/password,数据库的用户名和密码
...
canal.instance.dbUsername = root
canal.instance.dbPassword = 123456
#################################################

• start up

sh bin/startup.sh

• View server log

vi logs/canal/canal.log
---------------------------------
2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......

• View instance log

vi logs/example/example.log
-----------------------------------------
2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....

• shut down

sh bin/stop.sh

summary

--------The above are the installation steps of Canal.
Thank you for your support. If there is anything wrong, please give us feedback in time. Remember to like and support!