By Logstash synchronization data to a SQLServer Elasticsearch

Yan articles ELK spend the required environment, the new logstash profile

You need to link database driven Microsoft JDBC driver 6.2 for SQL Server

Download: https://www.microsoft.com/zh-CN/download/details.aspx?id=55539

In logstash the bin folder and add the following file folder jdbcconfig

New Logstash profile jdbc.conf

input {
    jdbc {
     jdbc_driver_library => "D:\ELK_logs\logstash-6.3.2\bin\jdbcconfig\mssql-jdbc-6.2.2.jre8.jar"
            jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
            jdbc_connection_string => "jdbc:sqlserver://192.168.100.51:1433;DatabaseName=BTPreservation;"
            jdbc_user => "sa"
            jdbc_password => "Rl123456"
                     # schedule => 分 时 天 月 年  
                        # schedule => * 22  *  *  *     //will execute at 22:00 every day
            schedule => "* * * * *"
            jdbc_paging_enabled => true
            jdbc_page_size => 1000
            clean_run => false
            use_column_value => true
            # Set the query field
              tracking_column => FID
            record_last_run => true
            last_run_metadata_path => "D:\ELK_logs\logstash-6.3.2\bin\jdbcconfig\FID.txt"
            # Set the column name lowercase
              lowercase_column_names => false
            statement_filepath => "D:\ELK_logs\logstash-6.3.2\bin\jdbcconfig\x_Loan_PreservationAdvanceList.sql"
            Index of type #
              type => "advancelist"
    }
}

output {
    elasticsearch {
        hosts => ["192.168.100.50:9200"]
        index => "advancelist"
        document_id => "%{FID}"
    }
    stdout {
        #codec => json_lines
        # Set the output format
        codec => line {
            format => "FID: %{[FID]} FPersonName: %{[FPERSONNAME]} FAddTime: %{[FADDTIME]}"
        }
    }
}

• Here is the biggest store down the last query ID obtained every time you perform this synchronization ID is set to query, access the incremental data, if we use the last update time as the judgment condition, you can get the latest data of the entire table

FID.txt stored query, automatically generates Once configured

 

x_Loan_PreservationAdvanceList.sql need to synchronize data performed Sql

SELECT * FROM dbo.x_Loan_PreservationAdvanceList WHERE FID > :sql_last_value

 

Note: Because logstash run multiple instances of the same machine, you need to specify a different data storage directory path.Data start

Excuting an order:

.\logstash -f .\jdbcconfig\jdbc.conf --path.data=/jdbcconfig/

 

After the success of the execution, creating an index kibana

 

Because when we execute the command, use the cmd window, when you add data to synchronize database table is synchronized in real time to see where the cmd window

Later we can also use NSSM tool to install it as a service running

Question: If we want to synchronize multiple tables how to deal with? What kind of scenarios require such synchronization.