elasticsearch5.2.1使用logstash同步mysql

centos 本人亲测可以首先安装好mysql，elasticsearch 不懂的请参考另一篇文章

安装logstash
官方：https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
1.下载公共密钥
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
2.添加yum源
vim /etc/yum.repos.d/logstash.repo
文件中写入
[logstash-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
保存退出
3.使用yum安装
yum install logstash
4.验证是否安装成功
进入 logstash 安装目录
cd /usr/share/logstash
运行
bin/logstash -e 'input { stdin { } } output { stdout {} }'
等待几秒钟出现
The stdin plugin is now waiting for input:
然后输入
hello world
得到类似的结果
2016-11-24T08:01:55.949Z bogon hello world

安装logstash-input-jdbc插件

1.安装 ruby 和 rubygems（注意：需要 ruby 的版本在 1.8.7 以上）
# yum install -y ruby rubygems
检查 ruby 版本：
# ruby -v
ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
替换国内的镜像
gem sources --remove http://rubygems.org/

gem sources -a http://gems.ruby-china.org/
验证是否成功
gem sources -l
修改Gemfile的数据源地址
whereis logstash # 查看logstash安装的位置，我的在 /usr/share/logstash目录

cd /usr/share/logstash
vim Gemfile
修改 source 的值为： "https://gems.ruby-china.org/"

vim Gemfile.jruby-1.9.lock

# 找到 remote 修改它的值为：https://gems.ruby-china.org/

开始安装

./bin/logstash-plugin install --no-verify logstash-input-jdbc

2.写配置文件开始同步 vi xxx.conf

input {
    jdbc {
      type => "user"
      jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test" 
      jdbc_user => "root"    
      jdbc_password => "123456"
      jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "1000"
      statement => "select * from user"
      schedule => "* * * * *"
    }

    jdbc {
      type => "task"
      jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test" 
      jdbc_user => "root"    
      jdbc_password => "123456"
      jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
      jdbc_driver_class => "com.mysql.jdbc.Driver"
      jdbc_paging_enabled => "true"
      jdbc_page_size => "1000"
      statement => "select * from task"
      schedule => "* * * * *"
    }
}

filter{
 mutate{
       remove_field => [ "@timestamp", "@version"]
 }
}

output {
   if [type] == "user" {
    elasticsearch {
        hosts  => ["192.168.33.111:9200","192.168.33.112:9200","192.168.33.113:9200"]
        index => "testindex"
        document_type => "user"
        document_id => "%{id}"
        user => "elastic"
        password => "changeme"
    }
   }
   if [type] == "task" {
    elasticsearch {
        hosts  => ["192.168.33.111:9200","192.168.33.112:9200","192.168.33.113:9200"]
        index => "testindex"
        document_type => "task"
        document_id => "%{id}"
        user => "elastic"
        password => "changeme"
    }
   }
}

执行：bin/logstash -f xxx.conf

ok 大功告成！！！

elasticsearch5.2.1使用logstash同步mysql

猜你喜欢