Elasticsearch synchronization mysql data

The mysqL elsearch synchronized to the data (sync introduced instead, when the data changes along with changes el data), there are two cases, a single table and the data table associated data.

0. jdbc driver jar package put in the bin directory

1.logstash configuration is particularly important

reference:

https://blog.csdn.net/qq_16436555/article/details/9136071

   {{JDBC INPUT 
　　　　　# address database, user, password 
        jdbc_connection_string => "JDBC: MySQL: // IP: 3306 / Test" 
        jdbc_user => "the root" 
        jdbc_password => "the root" 
　　　　　# driving position arranged 

        # here best path is an absolute path, the path depends on the line of command allows the directory 
        jdbc_driver_library => "Sync-conf / MySQL-Connector-the Java-6.0.6.jar" 
        jdbc_driver_class => "com.mysql.jdbc.Driver" 
　　　　　# tabs open inquiry 

        jdbc_paging_enabled => "to true" 
        jdbc_page_size => "50000" 

　　　　　　# Incremental synchronization of key decision

　　　　　　# Last_run_metadata_path whether to clear the record, if the query is true then start from scratch every time the equivalent of all database records

   　　 clean_run => false

 　 　　# Whether to record the value of a column, if record_last_run is true, you can customize field names that we need a table 
  　　  , use other fields to track, rather than time, at which point the parameter must be true. Otherwise, the default track is the timestamp value. 
 
    　　use_column_value => to true 
   　　 # use_column_value If true, configure this parameter. this parameter is the name of the field in a given database. Of course, the field must be incremental, data can be time database of this kind of 
   　　 tracking_column => create_time 
       # only two choices numeric, timestamp 
　　　　　tracking_column_type => "timestamp" 
    　　# whether to record the results of the last, if true, will put the last execution value to the field of tracking_column recorded, saved to the specified file last_run_metadata_path 
   　　 record_last_run => to true 
    # in the SQL statement wHERE mY_ID>: last_sql_value to which:. sql_last_value get is the value of the file 
    last_run_metadata_path => " /etc/logstash/run_metadata.d/my_info " 
    # whether to turn lowercase field names.
    # There is a small hint, if you deal with it before it had a data and a corresponding search needs in Kibana in, then it changed to true,
    # Because the default is true, and Kibana is case sensitive. ES accurate to say that it should be case-sensitive 
    lowercase_column_names => false 
    # SQL your position, of course, you can also write directly to SQL here. 
    = #statement> the SELECT * t the FROM tabeName the WHERE t.creat_time> : sql_last_value the Order by creat_time asc

　　 statement_filepath => "/etc/logstash/statement_file.d/my_info.sql" #sql file execution path, the best absolute path. 
    # Note: Load on SQL file is a text file that can be, there needs to be noted that a jdbc {} plug-in can only handle one SQL statement, 
    # If you have multiple SQL to handle the case, only re-establish a jdbc {} plugin. }

    # Set the listening interval of each field meaning points (from left to right), hours, days, months, years, all * Default is all updated every minute 
    Schedule => "* * * * *" 
# used to distinguish between input data source can be distinguished in the output where the output to 
        type => "User"   
    } 
} 
output { 
    # log output setting window 
    stdout { 
        CODEC => json_lines 
    } 
IF [type] == "User" {

    {elasticsearch 
        the hosts => [ "127.0.0.1:9200" ] 
        # Note that the index value does not support uppercase 
        index => "User" 
        #document_type set their own, is not provided, the default is DOC 
        #document_type => ""  # here the values from the query sql column name, self-configuration according to need  the document_id => "% {ID}"  }

}

}

Key:

The default configuration is not 1.tracking_column tracked timestamp, tracking_column with corresponding use_column_value => true

2. Set the position of the previous track record, it is open record_last_run => true,

When incremental synchronization, the change tracking data will be updated el, so the sql statement to add  : sql_last_value  the Order by tracking_column asc 

This will to  write the latest last_run_metadata_path file racking_column value