通过Logstash6.3.2的logstash-input-jdbc插件，同步MySQL数据到ElasticSearch6.3.2服务器

1、下载官方Logstash

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz

2、配置logstash-input-jdbc插件环境

#查看gem环境
gem

RubyGems is a sophisticated package manager for Ruby.  This is a
basic help message containing pointers to more information.

  Usage:
    gem -h/--help
    gem -v/--version
    gem command [arguments...] [options...]

  Examples:
    gem install rake
    gem list --local
    gem build package.gemspec
    gem help install

  Further help:
    gem help commands            list all 'gem' commands
    gem help examples            show some examples of usage
    gem help platforms           show information about platforms
    gem help <COMMAND>           show help on COMMAND
                                   (e.g. 'gem help install')
    gem server                   present a web page at
                                 http://localhost:8808/
                                 with info about installed gems
  Further information:
    http://guides.rubygems.org

显示出gem帮助信息，说明已经安装。

3、如果没有安装gem

yum install gem

4、gem源修改

gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org/

5、查看gem源

gem sources -l

6、修改logstash目录中的Gemfile文件

vim Gemfile
source "https://gems.ruby-china.org/"

7、修改logstash目录中的Gemfile.lock文件

vim Gemfile.lock
remote https://gems.ruby-china.org/

8、安装gem bundler

gem install bundler

9、安装logstash-input-jdbc插件

bin/logstash-plugin  install logstash-input-jdbc

10、上传数据库驱动文件mysql-connector-java-5.1.44.jar到logstash目录。

11、编写logstash-input-jdbc-mysql.conf文件

input {
  jdbc {
    jdbc_driver_library => "/usr/elasticsearch/mysql-connector-java-5.1.44.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://IP:3306/xxx"
    jdbc_user => "xxx"
    jdbc_password => "xxx"
    # sql 语句文件
    # statement_filepath => "filename.sql"
    statement => "SELECT * from sys_user_log where  record_date> :sql_last_value"
    jdbc_paging_enabled => "true"
    jdbc_page_size => "50000"
    type => "jdbc"
    tracking_column => "update_date"
    use_column_value => false
    # 设置监听间隔  各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
    schedule => "* * * * *"
  }
}


output {
  stdout {
    codec => json_lines
  }
  elasticsearch_http {
    host => "192.168.1.210:9200"
    index => "sys_user_log"
  }
}

插件Github地址

需要注意，网上给出的测试例子有问题，会让初学者刚开始使用误认为同步组件出错了。类似问题

12、执行同步

/bin/logstash  -f  logstash-input-jdbc-mysql.conf

Elasticsearch官方input-jdbc说明文档

参考