Mysql + canal + zookeeper环境搭建

一、mysql集群搭建

1. mysql基本环境

操作系统： Linux version 2.6.32-431.el6.x86_64

数据库：MySQL Community Server 5.7.20

主节点IP：10.60.81.157

主节点IP：10.60.81.158

从节点IP：10.60.81.159

2. 安装mysql

2.1．官网下载MySQL

mysql-5.7.20-1.el6.x86_64.rpm-bundle.tar

2.2. 三个节点都安装：

2.2.1．三个节点查看是否安装mysql

rpm -qa | grep mysql

rpm -e --nodeps mysql-libs-5.1.66-2.el6_3.x86_64 （有则删除）

2.2.2. 三个节点都安装mysql

tar –xvf mysql-5.7.20-1.el6.x86_64.rpm-bundle.tar

rpm -ivh mysql-community-common-5.7.20-1.el6.x86_64.rpm

rpm -ivh mysql-community-libs-5.7.20-1.el6.x86_64.rpm

rpm -ivh mysql-community-client-5.7.20-1.el6.x86_64.rpm

rpm -ivh mysql-community-server-5.7.20-1.el6.x86_64.rpm

2.3．设置密码

2.3.1. 编辑配置文件my.cnf

[mysqld]

server-id=157 #####这里不同

log-bin=mysql-bin ####开启二进制日志

log-slave-updates

binlog-format=ROW

sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

skip-external-locking

key_buffer_size = 384M

max_allowed_packet = 1M

table_open_cache = 512

sort_buffer_size = 2M

read_buffer_size = 2M

read_rnd_buffer_size = 8M

myisam_sort_buffer_size = 64M

thread_cache_size = 8

query_cache_size = 32M

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

#skip-grant-tables=1

# Disabling symbolic-links is recommended to prevent assorted security risks

symbolic-links=0

log-error=/var/log/mysqld.log

pid-file=/var/run/mysqld/mysqld.pid

2.3.2. 启动mysql

service mysqld start

2.3.3. 获取MySQL的临时密码

如果安装的是RPM包，则默认是/var/log/mysqld.log

2.3.4. 登陆并修改密码

使用默认的密码登陆:

mysql -uroot -p

alter user 'root'@'localhost' identified by 'Mysql@123456';

flush privileges;

3. 创建同步数据用户及授权用户

3.1．授权其他机器登陆

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Mysql@123456' WITH GRANT OPTION;

flush privileges;

3.2. 创建同步用户

grant replication slave on *.* to 'mytest'@'10.60.81.%' identified by 'mytest@123';

4. mysql主主配置

4.1. 配置两台主机及启动

查看两台主机（10.60.81.157、10.60.81.158）配置文件（my.cnf）除了server-id不一样，其他几乎一模一样。配置文件写好后，我们把两台服务器上的mysql服务器启动起来

4.2. 登录10.60.81.157的mysql中查看master状态

mysql> show master status;

+------------------+----------+--------------+------------------+-------------------+

+------------------+----------+--------------+------------------+-------------------+

| mysql-bin.000002 | 1290 | | | |

+------------------+----------+--------------+------------------+-------------------+

1 row in set (0.00 sec)

4.3. 登录10.60.81.158的msyql中，把10.60.81.157配置成自己的主

Change master to master_host='10.60.81.157',master_port=3306,master_user='mytest',master_password='mytest@123',master_log_file='mysql-bin.000002',master_log_pos=1290;

同上，查看10.60.81.158，然后登录10.60.81.157,把10.60.81.158，配置成自己的主，然后分别在两台机器的mysql中，启动slave

4.4. 启动10.60.81.158状态

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 10.60.81.158

Master_User: mytest

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.000003

Read_Master_Log_Pos: 1290

Relay_Log_File: sz280317-relay-bin.000009

Relay_Log_Pos: 367

Relay_Master_Log_File: mysql-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 1290

Relay_Log_Space: 743

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

Replicate_Ignore_Server_Ids:

Master_Server_Id: 158

Master_UUID: b805346c-54e0-11e8-baa8-005056a42d94

Master_Info_File: /var/lib/mysql/master.info

SQL_Delay: 0

SQL_Remaining_Delay: NULL

Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

Master_Retry_Count: 86400

Master_Bind:

Last_IO_Error_Timestamp:

Last_SQL_Error_Timestamp:

Master_SSL_Crl:

Master_SSL_Crlpath:

Retrieved_Gtid_Set:

Executed_Gtid_Set:

Auto_Position: 0

Replicate_Rewrite_DB:

Channel_Name:

Master_TLS_Version:

1 row in set (0.00 sec)

ERROR:

No query specified

可以看到 Slave_IO_Running: Yes
Slave_SQL_Running: Yes

然后在看10.60.81.158的状态：

mysql> show slave status \G;

*************************** 1. row ***************************

Slave_IO_State: Waiting for master to send event

Master_Host: 10.60.81.157

Master_User: mytest

Master_Port: 3306

Connect_Retry: 60

Master_Log_File: mysql-bin.000005

Read_Master_Log_Pos: 717

Relay_Log_File: sz280318-relay-bin.000010

Relay_Log_Pos: 6027

Relay_Master_Log_File: mysql-bin.000002

Slave_IO_Running: Yes

Slave_SQL_Running: Yes

Replicate_Do_DB:

Replicate_Ignore_DB:

Replicate_Do_Table:

Replicate_Ignore_Table:

Replicate_Wild_Do_Table:

Replicate_Wild_Ignore_Table:

Last_Errno: 0

Last_Error:

Skip_Counter: 0

Exec_Master_Log_Pos: 717

Relay_Log_Space: 7402

Until_Condition: None

Until_Log_File:

Until_Log_Pos: 0

Master_SSL_Allowed: No

Master_SSL_CA_File:

Master_SSL_CA_Path:

Master_SSL_Cert:

Master_SSL_Cipher:

Master_SSL_Key:

Seconds_Behind_Master: 0

Master_SSL_Verify_Server_Cert: No

Last_IO_Errno: 0

Last_IO_Error:

Last_SQL_Errno: 0

Last_SQL_Error:

Replicate_Ignore_Server_Ids:

Master_Server_Id: 157

Master_UUID: 1412180f-54c9-11e8-9e3e-005056a436f6

Master_Info_File: /var/lib/mysql/master.info

SQL_Delay: 0

SQL_Remaining_Delay: NULL

Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates

Master_Retry_Count: 86400

Master_Bind:

Last_IO_Error_Timestamp:

Last_SQL_Error_Timestamp:

Master_SSL_Crl:

Master_SSL_Crlpath:

Retrieved_Gtid_Set:

Executed_Gtid_Set:

Auto_Position: 0

Replicate_Rewrite_DB:

Channel_Name:

Master_TLS_Version:

1 row in set (0.00 sec)

ERROR:

No query specified

5．mysql主从配置

查看两台机器（10.60.81.157、10.60.81.159），把10.60.81.157作主，10.60.81.159作从，与主主配置唯一不同是，不要在主机配置从机为自己的主即可。

二、zookeeper集群搭建

1. zookeeper基本环境

操作系统： Linux version 2.6.32-431.el6.x86_64

Zookeeper: zookeeper-3.4.12.tar.gz

三台机器：10.60.81.160、10.60.81.161、10.60.81.162

2. zookeeper 安装与配置

2.1．官网下载zookeeper

zookeeper-3.4.12.tar.gz

2.2. 上传到远程节点

scp zookeeper.tar.gz [email protected]

2.3. 登录远程节点（其他两个节点的擦作大同小异）

ssh [email protected]

2.4. 解压包，配置，在cnof 目录新建zoo.cfg

sz280320:cloudera-scm@/hgx/zookeeper-3.4.12/conf>cat zoo.cfg

tickTime=2000

initLimit=5

syncLimit=2

dataDir=/hgx/servers/zookeeper/data

dataLogDir=/hgx/servers/zookeeper/logs

clientPort=4180

server.60=10.60.81.160:2888:3888

server.61=10.60.81.161:2888:3888

server.62=10.60.81.162:2888:3888

在目录上述dataDir的目录中创建myid, 将上述的三个数，server.X(我这里的X是60，61，62),中的X值写入myid文件，三台节点的myid这是不一样的。

3、zookeeper 启动与运行

3.1. 三个节点都要启动

zookeeper/bin/zkServer.sh start

3.2、zookeeper状态检查

sz280320:cloudera-scm@/hgx/zookeeper-3.4.12/bin>./zkCli.sh -server 10.60.81.160:4180

Connecting to 10.60.81.160:4180

2018-05-21 11:26:37,346 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT

2018-05-21 11:26:37,355 [myid:] - INFO [main:Environment@100] - Client environment:host.name=sz280320

2018-05-21 11:26:37,355 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.8.0_151

2018-05-21 11:26:37,361 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation

2018-05-21 11:26:37,361 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.8.0_151/jre

2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/hgx/zookeeper-3.4.12/bin/../build/classes:/hgx/zookeeper-3.4.12/bin/../build/lib/*.jar:/hgx/zookeeper-3.4.12/bin/../lib/slf4j-log4j12-1.7.25.jar:/hgx/zookeeper-3.4.12/bin/../lib/slf4j-api-1.7.25.jar:/hgx/zookeeper-3.4.12/bin/../lib/netty-3.10.6.Final.jar:/hgx/zookeeper-3.4.12/bin/../lib/log4j-1.2.17.jar:/hgx/zookeeper-3.4.12/bin/../lib/jline-0.9.94.jar:/hgx/zookeeper-3.4.12/bin/../lib/audience-annotations-0.5.0.jar:/hgx/zookeeper-3.4.12/bin/../zookeeper-3.4.12.jar:/hgx/zookeeper-3.4.12/bin/../src/java/lib/*.jar:/hgx/zookeeper-3.4.12/bin/../conf:

2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp

2018-05-21 11:26:37,362 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA>

2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux

2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64

2018-05-21 11:26:37,363 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64

2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.name=cloudera-scm

2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/home/cloudera-scm

2018-05-21 11:26:37,364 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/hgx/zookeeper-3.4.12/bin

2018-05-21 11:26:37,368 [myid:] - INFO [main:ZooKeeper@441] - Initiating client connection, connectString=10.60.81.160:4180 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29

Welcome to ZooKeeper!

2018-05-21 11:26:37,430 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@1028] - Opening socket connection to server 10.60.81.160/10.60.81.160:4180. Will not attempt to authenticate using SASL (unknown error)

JLine support is enabled

2018-05-21 11:26:37,622 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@878] - Socket connection established to 10.60.81.160/10.60.81.160:4180, initiating session

2018-05-21 11:26:37,653 [myid:] - INFO [main-SendThread(10.60.81.160:4180):ClientCnxn$SendThread@1302] - Session establishment complete on server 10.60.81.160/10.60.81.160:4180, sessionid = 0x3c039e8bf59a0003, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

三、搭建HA模式的canal

1. canal基本环境

操作系统： Linux version 2.6.32-431.el6.x86_64

canal server： canal.deployer-1.0.25.tar.gz

二台机器：10.60.81.159、10.60.81.158

2．Canal安装

2.1. 官网下载Canal

canal.deployer-1.0.25.tar.gz

2.2. 新建canal用户，并赋予权限

CREATE USER canal IDENTIFIED BY 'canal';

GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';

FLUSH PRIVILEGES;

（在10.60.81.157机器中进入mysql创建canal用户，并赋予权限）

2.3. 上传包到两个远程节点

scp canal.deployer-1.0.25.tar.gz [email protected]:/hgx/

2.4．二个节点安装canal

tar –zxvf canal.deployer-1.0.25.tar.gz

mv canal.deployer-1.0.25 canal

3. canal配置文件

3.1. canal.properties

#################################################

######### common argument #############

#################################################

canal.id= 1

canal.ip=

canal.port= 11111

canal.zkServers=10.60.81.160:4180,10.60.81.161:4180,10.60.81.162:4180

# flush data to zk

canal.zookeeper.flush.period = 1000

# flush meta cursor/parse position to file

canal.file.data.dir = ${canal.conf.dir}

canal.file.flush.period = 1000

## memory store RingBuffer size, should be Math.pow(2,n)

canal.instance.memory.buffer.size = 16384

## memory store RingBuffer used memory unit size , default 1kb

canal.instance.memory.buffer.memunit = 1024

## meory store gets mode used MEMSIZE or ITEMSIZE

canal.instance.memory.batch.mode = MEMSIZE

## detecing config

canal.instance.detecting.enable = false

#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()

canal.instance.detecting.sql = select 1

canal.instance.detecting.interval.time = 3

canal.instance.detecting.retry.threshold = 3

canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery

canal.instance.transaction.size = 1024

# mysql fallback connected to new master should fallback times

canal.instance.fallbackIntervalInSeconds = 60

# network config

canal.instance.network.receiveBufferSize = 16384

canal.instance.network.sendBufferSize = 16384

canal.instance.network.soTimeout = 30

# binlog filter config

#canal.instance.filter.druid.ddl = true

canal.instance.filter.query.dcl = false

canal.instance.filter.query.dml = false

canal.instance.filter.query.ddl = false

canal.instance.filter.table.error = false

canal.instance.filter.rows = false

# binlog format/image check

canal.instance.binlog.format = ROW,STATEMENT,MIXED

canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation

canal.instance.get.ddl.isolation = false

#################################################

######### destinations #############

#################################################

canal.destinations= kafka

# conf root dir

canal.conf.dir = ../conf

# auto scan instance dir add/remove and start/stop instance

canal.auto.scan = true

canal.auto.scan.interval = 5

#canal.instance.tsdb.spring.xml=classpath:spring/tsdb/h2-tsdb.xml

#canal.instance.tsdb.spring.xml=classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring

canal.instance.global.lazy = false

#canal.instance.global.manager.address = 127.0.0.1:1099

#canal.instance.global.spring.xml = classpath:spring/local-instance.xml

#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml

#canal.instance.global.spring.xml = classpath:spring/file-instance.xml

canal.instance.global.spring.xml = classpath:spring/default-instance.xml

3.2. instance.properties

#################################################

## mysql serverId

canal.instance.mysql.slaveId=1234 ##另外一台机器改成1235，保证slaveId不重复即可

# position info

canal.instance.master.address=10.60.81.157:3306

canal.instance.master.journal.name=

canal.instance.master.position=

canal.instance.master.timestamp=

# table meta tsdb info

canal.instance.tsdb.enable=true

canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}

canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;

#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb

canal.instance.tsdb.dbUsername=canal

canal.instance.tsdb.dbPassword=canal

#canal.instance.standby.address =

#canal.instance.standby.journal.name =

#canal.instance.standby.position =

#canal.instance.standby.timestamp =

# username/password

canal.instance.dbUsername=canal

canal.instance.dbPassword=canal

canal.instance.defaultDatabaseName=

canal.instance.connectionCharset=UTF-8

# table regex

canal.instance.filter.regex=.*\\..*

# table black regex

canal.instance.filter.black.regex=

#################################################

***上述两个配置文件，黄色标记部分为修改部分在两个服务器上一样***

4. 启动canal服务

两台机器分别执行：canal/bin/zkServer.sh

5. 查看是否启动成功

5.1. 在10.60.81.159查看日志

5.2. 在10.60.81.158查看日志

***上述两服务器日志文件(canal.log)出现如图所示，说明canal的HA搭建成功