In most cases our data are stored in the database, but elasticsearch it has its own index database, so if we do a search is when you need the data in the database in sync to elasticsearch, where we use the logstash logstash-jdbc-input plug-ins with database synchronization, database synchronization and for logstash, we can set the time elasticsearch synchronized with the database, the use of this approach to synchronization is still very convenient.
1. Download and install logstash
attention to the downloaded version and the version number of your elasticsearch consistent, my version elasticsearch6.3.0
logstash Download: https://www.elastic.co/downloads/logstash
Once downloaded, unzip directly
2. Configure logstash
for logstash5.x or later, it has integrated the plugin itself, we do not need to be installed separately, can be used directly. I am here to talk about the simple configuration and synchronization mysql, in logstash file directory, create a new folder (named freely). Such as: mysql
2.1. A first jdbc driver into this folder, used to connect mysql database
2.2 Create a .conf configuration file (random name), is used to associate with the database es, here is mysql.conf
{the INPUT
jdbc {
of the type => "tb_user"
# MySQL database links, shop for the database name
jdbc_connection_string => "jdbc: MySQL: //192.168.1.231: 3306 / dcp_test"
# username and password
jdbc_user => "gyadmin"
jdbc_password => "!! Gy (DJZ 159"
# driver
jdbc_driver_library => "F: /elkStudy/logstash/logstash-6.2.3/mysql/mysql-connector-java-5.1.42.jar"
# driver class name
jdbc_driver_class => "com .mysql.jdbc.Driver "
jdbc_paging_enabled =>" to true "
jdbc_page_size =>" 50000 "
SQL file path # execution + name
statement_filepath =>" F: /elkStudy/logstash/logstash-6.2.3 / MySQL / jdbc.sql "
# Set the listening interval of each field meaning points (from left to right), hours, days, months, years, all * Default is all updates every minute
= Schedule> "* * * * *"
}
jdbc {
of the type => "tb_tree"
# MySQL database links, shop for the database name
jdbc_connection_string => "jdbc: MySQL: //192.168.1.231: 3306 / dcp_test"
# username and password
= jdbc_user> "gyadmin"
jdbc_password => "!! Gy (DJZ 159"
# driver
jdbc_driver_library => "F: /elkStudy/logstash/logstash-6.2.3/mysql/mysql-connector-java-5.1.42.jar"
# driver class name
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "to true"
jdbc_page_size => "50000"
# Sql file path of execution + name
statement_filepath => "F: /elkStudy/logstash/logstash-6.2.3/mysql/jdbc_tree.sql"
# Set points describes the fields listening interval (from left to right), hours, days, months, years, all * Default is updated every minute are
Schedule => "* * * * *"
}
}
filter {
JSON {
Source => "Message"
remove_field => [ "Message"]
}
}
Output {
IF [type] == "tb_user" {
elasticsearch {
the IP address and port # ES of
the hosts => [ "192.168.3.94:9200"]
# index name
index => "mysql_tb_user"
database # has a need to be correlated id field, id corresponding to the type of
the document_id => "% {id}"
# document type
document_type => "user"
}
}
if [type]=="tb_tree" {
{elasticsearch
# ES IP address and port of
the hosts => [ "192.168.3.94:9200"]
# index name
index => "mysql_tb_tree"
database associated required # id field has a corresponding type of id
the document_id => " {ID}% "
# document type
DOCUMENT_TYPE =>" Tree "
}
}
stdout {
# outputs the JSON format
CODEC => json_lines
}
}
2.3 Creating sql file (random name), as in the present embodiment and jdbc.sql jdbc_tree.sql. Note: sql can not have at the end, otherwise the time will run error (As for the reason, will be mentioned later)
jdbc.sql:
SELECT * FROM tb_user
jdbc_tree.sql
SELECT * FROM tb_tree
3. Start logstash
in the bin directory logstash using cmd execute the command: logstash -f ../mysql/mysql.conf, you will see the following information
Can be seen in the synchronization process, the implementation of our script, and it is wrapped, it can not have a sql script terminator. Will drop to synchronize data printed json string out of the way (the default field name all lowercase a)
4. Check data elasticsearch-head
You can see all the data has been synchronized to the elasticsearch. However, in the default index increased @version and @timestamp two fields
5. stepped pit
logstash initiate a message can not find the main class solutions, refer to: https://www.cnblogs.com/sbj-dawn/p/8549369 .html
6.Q & a
1, new or updated database data, we find that logstash will be based on a set time (here I set every minute, is the minimum frequency) to automatically sync to the latest data es.
2, logstash currently only supports incremental data, incremental table, that synchronization is not physically delete the data and tables. The solution to this problem may reference https://blog.csdn.net/laoyang360/article/details/51747266