Windows使用go-mysql-elasticsearch工具同步Mysql数据到ElasticSearch
go-mysql-elasticsearch简介
1.go-mysql-elasticsearch是一个将MySQL数据自动同步到Elasticsearch的服务。它首先使用mysqldump获取原始数据，然后用binlog增量地同步数据。
2. 这里有几点注意事项：
Mysql的binlog必须是ROW模式，不然启动会报错。
连接Mysql的用户权限需要大一些。
安装
1.Go安装包下载
#Windows安装
https://studygolang.com/dl //Go下载地址
#打开Cmd
#安装godep
go get github.com/tools/godep
#下载go-mysql-elastisearch插件
go get github.com/siddontang/go-mysql-elasticsearch
#切换目录到
C:\Users\MarLonBrando\go\src\github.com\siddontang\go-mysql-elasticsearch\cmd\go-mysql-elasticsearch
#编译
go build
#linux获取安装包
yum install -y go
#安装godep
go get github.com/tools/godep
#下载go-mysql-elastisearch插件
go get github.com/siddontang/go-mysql-elasticsearch
#进入对应目录，比如我使用的如下目录/root/go/src/github.com/siddontang/go-mysql-elasticsearch
cd /root/go/src/github.com/siddontang/go-mysql-elasticsearch
#编译
make
Windows编译成功后会返回一个.exe文件如下
2.配置开启binlog
- Windows
需要在mysql中开启binlog，首先查询一下是否开启了binlog
Windows在打开my.ini文件，在mysqld下面添加,保存文件，重启mysql服务
log_bin=mysql-bin
binlog-format=ROW
server-id=1
- linux
接下来需要在mysql中开启binlog，首先查询一下是否开启了binlog。
#进入mysql
mysql -uroot -p
#输入密码，然后输入如下命令查看binlog开启状态
show variables like '%log_bin%';
#，ON为开启了，如果没有开启的话为OFF。
如果没有开启的话，需要在my.cnf配置中添加如下配置(其中server-id可以根据情况设置，这里设置为1，log-bin为日志位置，一定要给日志写的权限，不然会报错，binlog_format为模式，这里必须为ROW)：
# MySQL address, user and password
# user must have replication privilege in MySQL.
my_addr = "127.0.0.1:3306"
my_user = "root"
my_pass = "1234"
my_charset = "utf8"
# Set true when elasticsearch use https
#es_https = false
# Elasticsearch address
es_addr = "127.0.0.1:9200"
# Elasticsearch user and password, maybe set by shield, nginx, or x-pack
es_user = ""
es_pass = ""
# Path to store data, like master.info, if not set or empty,
# we must use this to support breakpoint resume syncing.
# TODO: support other storage, like etcd.
data_dir = "./var"
# Inner Http status address
stat_addr = "127.0.0.1:12800"
stat_path = "/metrics"
# pseudo server id like a slave
server_id = 1001
# mysql or mariadb
flavor = "mysql"
# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump = "mysqldump"
# if we have no privilege to use mysqldump with --master-data,
# we must skip it.
#skip_master_data = false
# minimal items to be inserted in one bulk
bulk_size = 128
# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time = "200ms"
# Ignore table without primary key
skip_no_pk_table = false
# MySQL data source
[[source]]
# 需要同步的数据库
schema = "bigdatacar"
# Only below tables will be synced into Elasticsearch.
# "t_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023
# I don't think it is necessary to sync all tables in a database.
#tables = ["t", "t_[0-9]{4}", "tfield", "tfilter"]
tables = ["t_user"]
[[rule]]
schema = "bigdatacar"
table ="t_user"
index = "bigdatacar"
type = "t_user"
[rule.field]
# Map column `id` to ES field `es_id`
id="es_id"
# Map column `tags` to ES field `es_tags` with array type
tags="es_tags,list"
# Map column `keywords` to ES with array type
keywords=",list"
# Only sync following columns
#filter = ["id", "name"]
# The es doc's id will be `id`:`tag`
# It is useful for merge muliple table into one type while theses tables have same PK
#id = ["id", "tag"]
运行程序进行数据同步
C:\Users\MarLonBrando\go\src\github.com\siddontang\go-mysql-elasticsearch\cmd\go-mysql-elasticsearch>
go-mysql-elasticsearch.exe -config=C:\Users\MarLonBrando\go\src\github.com\siddontang\go-mysql-elasticsearch\etc\river.toml
Es查看索
http://localhost:9200/_cat/indices?v
