Benpian chapter address: https: //www.cnblogs.com/Thehorse/p/11601013.html
Today we are speaking about logstash synchronization mysql data to es
I think of it, logstash is one of many synchronization mysql data to plug-es, the most stable and easiest to configure one.
input {
stdin {
}
jdbc {
type => "xxx"
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/yinhelaowu"
jdbc_user => "root"
jdbc_password => "root"
jdbc_driver_library => "mysql1/mysql-connector-java-5.1.17-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => true
jdbc_page_size => "500"
record_last_run => true
use_column_value => true
tracking_column => "id"
last_run_metadata_path => "mysql1/jdbc1"
clean_run => "false"
statement_filepath => "mysql1/jdbc.sql"
schedule => "* * * * *"
clean_run => false
}
}
I am speaking about the role of these attributes, like type this is mainly to declare the jdbc {} This process objects, type can define any value,
jdbc_connection_string, this is the address of the database as well as user name and password
jdbc_driver_library, logstash is required to connect mysql jdbc and java environment must be configured using logstash, jdk1.8 above
The connection to jdbc_driver_class named statement_filepath this sql statement is executed
How long is a schedule synchronization at the data if it is ****** represents one minute synchronization time
last_run_metadata_path This will record some of the values you last synchronized data, for example, you can create a record time, and auto-incremented id,
type index type
Clean_run clear whether the data have been executed
To the bottom
# This is the output stream
output {
#这个type是上面jdbc的type,有了这个if就可以进行批处理
if [type]=="TInquiryInfo"{ elasticsearch {
#这边的端口一定是9200，java中使用的端口是9300这边不要弄混了 hosts => ["localhost:9200"] # 索引名称 index => "ca-inquiry" # type名称 document_type => "TInquiryInfo" # 文档id，inquiryId为sql文件中查询出的字段名 document_id => "%{inquiryId}" } } }
下面是执行的sql写法
SELECT c.*,CONCAT(CONCAT(c.lat,','),c.lng) AS location,u.`praise`,u.`age`,u.`area_id` FROM zsf_carftsmanship c
LEFT JOIN zsf_user u ON c.user_id = u.`id` WHERE c.id > :sql_last_value
Like elasticsearch such a large amount of data query, multi-table associated with the query undoubtedly have a great impact on performance, so I'm here to do directly related queries
So can greatly improve query performance es of this: sql_last_value value is recorded in last_run_metadata_path