Installation is simple and practical Logstash (data to synchronize MySql elasticsearch)

 

 

Logstash is a lightweight log collection processing framework, it can easily scattered and diverse gathering up logs, and custom processing and then transferred to the specified location, such as a server or a file

Windows environment:

　　1, download logstash

　　Address:  https://www.elastic.co/cn/downloads/logstash

　　2. Download: mysql-connector-java.jar

　　Create a mysql directory under logstash directory, the driver files into them.

　　2, modify the configuration file

　　In the installation directory, create a new configuration file named: mysqltoes.conf　

　　This file name can be replaced with other, startup script commands into corresponding on it.

　　Add the following to the file:

the INPUT { 
  
  # multiple tables synchronized only need to set up multiple jdbc module on the line 
  jdbc { 
      # MySQL database links, shop for the database name 
      jdbc_connection_string => "jdbc: MySQL: // database IP address:? 3306 / test useUnicode = true & characterEncoding UTC = utf8 & serverTimezone = " 
      # username and password 
      jdbc_user => " root " 
      jdbc_password => " " 

      # driver 
      jdbc_driver_library => " E: /DevelopEnvironment/Logstash/logstash-7.2.0/mysql/mysql-connector-java-8.0.16 .jar " 

      # driver class name 
      jdbc_driver_class => " com.mysql.jdbc.Driver " 

      # whether paging 
      jdbc_paging_enabled =>"true"
      jdbc_page_size => "50000" 

      # direct implementation of sql statement 
      of Statement => "from the Employee the SELECT *" 
      sql file path # execution + name 
      # statement_filepath => "/hw/elasticsearch/logstash-6.2.4/bin/test.sql" 

      # set the listening interval of each field meaning points (from left to right), hours, days, months, years, all * default is all updated every minute 
      Schedule => "* * * * *" 

      # index type 
      #type => "JDBC" 
    } 

} 


Output { 
  elasticsearch { 
        #es ip and port of 
        the hosts => [ "HTTP: // ip: 9200"] 
        #ES index name (their definition)
            index => "Blog" 
        # Document Type 
        DOCUMENT_TYPE => "log" 
        ID # Set data field in the database for 
        the document_id => "% {ID}" 
    } 
    stdout { 
        CODEC => json_lines 
    } 

}

 

You need to configure:

Database address, account password database, database-driven path, query, regular scanning time

es address, index, type, primary key field

Once configured, open cmd window, type start command: bin \ logstash.bat -f config \ mysqltoes.conf as shown below:

If you can not find the file, check the command path and file name.

 

Run successfully: