logstash synchronization mysql data to Elasticsearch Docker deployment logstash

Installation logstash see my other article   Docker deployment logstash

Synchronous data we first need to install the corresponding plug-ins, and then download the corresponding jar package database link, the following are specific steps

 

A mounting logstash-input-jdbc

1, into the container

docker  exec  it  logstash  bash

2, into the bin directory, I have here is / usr / share / logstash / bin, you can see logstash-plugin file, and then install the plug

logstash-plugin install logstash-input-jdbc

3, see the following output, the installation successful 

 

Two, logstash to synchronize data Es mysql

1, Introduction to the scene such as we need to retrieve news article, simply use mysql to achieve efficiency is too low, especially when large volumes of data. This time we can use es, logstash timing of the new and updated articles to synchronize es, we can call the API to retrieve articles es directly on business.

2, the configuration file and jdbc.sql jdbc.conf file in conf.d directory

(1) Profile jdbc.conf

{INPUT 
    stdin {} 
    JDBC { 
　　　　　database address # connection and a database which specifies the encoding format, disable the SSL protocol, set automatically reconnect　
        jdbc_connection_string => " JDBC: MySQL: //192.168.16.241:? 3306 / wsy_blockchain characterEncoding = = & useSSL. 8-UTF to false to true & autoReconnect = & serverTimezone the UCT = " 
        jdbc_user => " the root " 
        jdbc_password => " 12345678 " 
        # mysql connecting a jar storage path 
        jdbc_driver_library => " /opt/mysql-connector-java-8.0.15.jar " 
        # drive 
        jdbc_driver_class => " COM.mysql.cj.jdbc.Driver" 
        Jdbc_paging_enabled => " to true " 
        jdbc_page_size => " 50000 " 
        CODEC => Plain { 
            charset => " UTF-8 " 
        } 
        # track field I use is to create time (mainly used to test) 
        tracking_column => CreateTime 
        record_last_run = > to true 
        Why # for this reason could not find the error will be open 
        last_run_metadata_path => " /usr/local/opt/logstash/lastrun/.logstash_jdbc_last_run " 
        reason # Why not find this error will be open
        #jdbc_default_timezone => " Asia / on Shanghai "         　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　 statement # synchronous data storage location　　　　　　　　　　　　　
        statement_filepath => " /usr/share/logstash/bin/jdbc.sql " 
        # last_run_metadata_path whether to clear the record, if the query is true then start from scratch every time the equivalent of all database records 
        clean_run => to false 

        # this is a control timing, the time interval is repeatedly executed tasks introduced, the first one is not provided minutes is 1 minute to perform a 
        Schedule => " * * * * * " 
        type => " STD " 
    } 

} 

filter { 
    JSON { 
        Source  =>" Message "
        remove_field => ["message"]
    }
    ruby { 
        code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" 
    }
    ruby {
        code => "event.set('@timestamp',event.get('timestamp'))"
    }
    mutate {
        remove_field => ["timestamp"]
    }
  
}

output {

    elasticsearch {

        hosts => "192.168.16.241:9200"

        index => "test_data"

        document_type => "users"

        document_id => "%{id}"
    }

    stdout {

        codec => json_lines

    }
}        

 

(2) Configuration jdbc.sql

select id,nickname,mobile, createtime  from  users where createtime >  :sql_last_value

3, the startup configuration file jdbc.conf files begin synchronizing data

logstash -f  jdbc.conf

4, you can see a synchronized start

 

Two major problems encountered

1, check out the database users to create less time 8 hours

Solution: database configuration parameters need to add links serverTimezone = UCT

2, a time stamp synchronous data less 8 hours,

Resolution: The synchronization filter corresponding field timestsmp, plus eight hours after obtaining, the following

filter {
    ruby { 
        code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)" 
    }
    ruby {
        code => "event.set('@timestamp',event.get('timestamp'))"
    }
    mutate {
        remove_field => ["timestamp"] }  }