verification of the acquired incremental logstash mysql

    A recent demand data needed to mysql in a table are synchronized to the es, the data using logstash after analysis jdbc plug-in or take in mysql, output to es, the situation is collected in two ways: the beginning of the whole amount of the acquisition, after the incremental acquisition.

Verification process has the following formula:

1. Install and mysql lonstash

mysql and logstash installation is not to say, you can find one online.

Things to note:

    1) may need to install: bin / plugin logstash the install-INPUT-JDBC

    2) mysql download a driver https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz

2. Create mysql tables and data

create table test.zy  (
id int,
str varchar(20)
) ;

insert into test.zy   values('1','a1');
insert into test.zy   values('2','a2');
insert into test.zy   values('3','a3');
insert into test.zy   values('4','a4');
insert into test.zy   values('5','a5');
insert into test.zy   values('6','a6');
insert into test.zy   values('7','a7');
insert into test.zy   values('8','a8');
insert into test.zy   values('9','a9');
insert into test.zy   values('10','a10');
insert into test.zy   values('11','a11');
insert into test.zy   values('12','a12');
insert into test.zy   values('13','a13');
insert into test.zy   values('14','a14');
# Increment data acquisition verification inserted
insert into test.zy   values('15','a15');
insert into test.zy   values('16','a16');

3.logstash collection mysql configuration file

According to the incremental acquisition field #
input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://ip:3306/test"
        jdbc_user => "root"
        jdbc_password => "123456"
        jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        statement => "select * from zy where id > :sql_last_value"
	        use_column_value => true
	        tracking_column => "id"
	        record_last_run => true
	        last_run_metadata_path => "/root/test.log"
	        schedule => "*/2 * * * *"
     }
}

output {
  file {
   path => "./mysql/test-%{+YYYY-MM-dd}.txt"
  }
}

# Collected according to the timestamp increment
input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql://ip:port/test"
        jdbc_user => "root"
        jdbc_password => "123456"
        jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_default_timezone => "Asia / Shanghai" # Set sql_last_value recording time zone, or it will affect the acquisition of the incremental effect
        statement => "select * from zy1 where time > :sql_last_value"
        use_column_value => false
        record_last_run => true
        last_run_metadata_path => "/root/test.log"
        schedule => "*/2 * * * *"
     }
}

output {
  file {
   path => "./db2/test-%{+YYYY-MM-dd}.txt"
  }
}

The configuration parsing input
statement execution myqsl statement may also be followed by statementpath sql file path
use_column_value 
    Whether to use column values ​​as the basis for recording the location of the last run.
    If set to true, the tracking_column columns defined as: sql_last_value.
    If set to false, then: sql_last_value reflect the previous SQL running time.
tracking_column incremental acquisition is based on the field names if use_colomn_value is false, you can not write
Whether the current recording position record_last_run data collection
last_run_metadata_path set record file data collection location
Frequency schedule sql script execution

4. Verify acquisition

    Start logstash

    cd /usr/local/logstash

    bin/logstash -f conf/logtash.conf

    logstash output log acquisition:

    

   

View the contents of the output file

sql_last_value record