Elasticsearch use of Quick --Spring Boot Elastcisearch, using Logstash mysql synchronous data and Elasticsearch

If it is not Spring Boot item, then select the corresponding version of dependence according to Elasticsearch version.

Example: Elasticsearch version is 5.4.0, then

　　　　<dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>5.4.0</version>
        </dependency>

Spring Boot below began to introduce integrated Elasticsearch, and how to synchronize data and Elasticsearch of mysql.

My version (through my own use, make sure there is no problem):

　　1. Computer: windows10 operating system

　　2.Elasticsearch Version: 7.0.0

　　3.ik word version: 7.0.0 (must be consistent and Elasticsearch version)

　　4.postman: None Version (indexs to view this data), of course, there is a similar tool curl, kibana, then the two versions need to be consistent and Elasticsearch, compared trouble (foreign Web sites, I usually spend on download 1 day before is finished, so do not want to download the two can choose not to download, for us to write the code has no effect)

　　5.elasticsearch-head-master: After the free version (download the source code to compile this, start-up mode:

　　　　We need to start Elasticsearch (browser, enter localhost: 9200, json text information appears on elasticsearch description of success), then:

　　　　Way 1.cmd npm run start after switching to a folder, and then enter localhost browser at URL: 9100

　　　　2. Select the way or simply double-click index.html file folder)

　　6.Spring Boot version: 2.2.4RELEASE

　　If some basic installation and use on Elasticsearch will not be Baidu, following the official start

 

Getting started: integrated way I understand there are several

　　1. elasticsearch official logstash be mysql and Elasticsearch data synchronization, I introduced is this. Limitations: the need and the same version of Elasticsearch version Logstash, Elasticsearch official website can download Logstash (download speed is very slow, I downloaded 7.0.0, more than 100M, spent a morning time)

　　2. Use Bboss this open source project, its official statement is compatible with all versions of Elasticsearch. So there is no version of the problem, you can click to start learning visit, describing in great detail

 　　3. Use elasticsearch-river-jdbc, not recommended. And there are two other similar species, but is not recommended, because: support Elasticsearch too low. This seems to be up to about 6.0, one of the other two only support about Elasticsearch2.0 version, so I remember not introduced, are interested can go and see.

 

Spring Boot using Elastcisearch, using Logstash mysql synchronous data and Elasticsearch:

1.Spring Boot使用Elasticsearch：

    <dependency> 
            <the groupId> org.springframework.boot </ the groupId> 
            <the artifactId> Starter-Spring-Boot-Data-elasticsearch </ the artifactId> 
        </ dependency>    
<-! versions are synchronized and parent Spring Boot default 2.2.4RELEASE, spring-data-lasticsearch version of which is 3.2.4.RELEASE, 
elasticsearch-REST-High-Level-Client and which is re-transport version 6.8.6 ->

 Specific integration may explain the reference in this article

2. Download Logstash, I chose the zip compressed download, unzip the package, can not put Chinese path (as developed with all the best things that put the English route, the last time I also encountered the installation path will be signed error of).

1. determine whether the installation is successful: Command forget, this is not important, we direct the next step.

2. Go Logstash the bin directory in which to create your own jdbc.conf file, the file name at random, behind cmd command to change along with the line: Note: I have a few lines configured hit # is a problem because these lines, I run after troubleshooting annotated:

{INPUT 
  stdin} { 
  jdbc { 
  #mysql the connection means jdbc jar package 
    jdbc_driver_library => "D: \ softwareRepository \ logstash-7.0.0 \ config \-Test config \ MySQL-Connector-Java-5.1.46.jar" 
    #jdbc driving class full class name 
    jdbc_driver_class => "com.mysql.jdbc.Driver" 
    #jdbc connection url 
    jdbc_connection_string => "jdbc: MySQL: // localhost: 3306 / the Test useUnicode = to true & characterEncoding = UTF-8 & useJDBCCompliantTimezoneShift = to true & useLegacyDatetimeCode = false & serverTimezone = UTC? " 
    # database user name 
    jdbc_user =>" root " 
    # database password 
    jdbc_password =>" root "
    # Data synchronization time and that (is * the synchronized once every minute) 
    Schedule=> "* * * * *" 
    #Jdbc paging query open 
    #jdbc_paging_enabled => "to true" 
    # query the number of data per page 
    #jdbc_page_size => "50000" 
    #sql file path (is the need to synchronize data) 
    statement_filepath => "D: \ softwareRepository \ logstash-7.0.0 \ config \ the Test-config \ zyl.sql" 
    # last updated position mark file path 
    last_run_metadata_path => "D: \ softwareRepository \ logstash-7.0.0 \ config \ the Test-config \ jobLastInfo.txt " 
    # whether to clear each time you start on a profile data 
    clean_run =>" false " 
    # turn all the field names to lower case 
    lowercase_column_names =>" to true " 
    # solve the Chinese garbled 
    CODEC => Plain charset = { > "UTF-8"} 
    record whether the record was last run #
    record_last_run => "To true" 
    # determines whether the other fields for data synchronization 
    use_column_value => "to true" 
    incremental index # database field name 
    tracking_column_type => "timestamp" 
    tracking_column => "UPDATE_TIME" 
  } 
} 

filter { 
    JSON { 
        Source => "Message" 
        remove_field => [ "Message" ] 
    } 
} 

Output { 
    elasticsearch { 
        index => "zyltest" 
        DOCUMENT_TYPE => "%} {type" 
        #es server 
        the hosts => "localhost: 9200"
        document_id => "%{id}"
    }
    stdout {
        codec => json_lines
    }
}

I made reference to the above configuration of three articles came to understand what it means: ...... they did not write the whole, especially .sql file is a problem, I now explain below:

After the files are copied, we went to the config directory, create a folder test-config (the file name to create their own things are casual, with the configuration file changes it wants to), placed inside a jobLastInfo.txt, zyl.sql, and a mysql-connector-java jar package. .Txt file which I do not know what effect, created trying to be difficult, and most critical is .sql file, which is a query, for example,

select * from test1;

I see the two articles did not mention what sql file written statements ...... I only tested a query, fast 23:00, tomorrow this article probably will not change, and you can see for yourself the query is executed two tables. And then a jar package, I started logstash reported the error on the jar package, and asked me to determine the set position jar of it (seems to be the problem)? But in fact there is no problem, still running.

I put out my database: create tables and insert first execution sentence on it, additions and deletions to step 3 appears as shown in the picture and then performed (and then look at the results in 00 seconds per minute, will find that although all changes the contents are Logstash check out, but only two additions and changes to the index update, delete no effect, and modify data even without update_time modifications will update the index )

drop table test1;
create table test1(
  id int primary key auto_increment,
  name varchar(10) default "我是笨蛋5号我！…",
  update_time date NOT NULL DEFAULT "2017-3-3 3:3:3"
);
insert into test1() value ();
update test1 set name="bb" where id in (1,2);
update test1 set name="cc",update_time=now() where id in(3,4);
update test1 set name="ddd" where id in (5,6);
delete from test1 where id in (7,8);

Point to note: tracking_column_type => "timestamp" This value must be a timestamp, or will be error to remind you to come back, but the database does not need to be a certain type tiemstamp of time, I tried datetime, date, time, timestamp four time type I have tried, the results are the same success . (Because either type, although the difference in the format database is not the same display, but the display type is Logstash difference data at the time that the format of FIG.).

3. Start elasticsearch, and you can choose to start postman elasticsearch-head-master to observe the situation. Started logstash: cmd switch to the bin directory execute: logstash -f jdbc.conf. FIG After Run Effect:

4. If you and I have exactly the same configuration, it will localhost: 9100 saw a zyltest the index generated by the postman GET to http: // localhost: 9200 / zyltest / _search can view the generated index (amount , forget what it meant, so be it).

5. Note: logstash can only increase simultaneously and to change, delete and therefore needs its own code is synchronized by deleteXX method (forget what the method, not looking up). In other words: in the open Elasticsearch and Logstash service, it will increase and synchronize changes automatically every minute, and then synchronize the data to manually delete the code.

This is the end

 

Forehead, writes the next half switching wifi, it did not save the contents of the draft, auto-refresh interface is gone, tired ......