将mysql数据同步到elasticsearch

一、同步mysql数据，显示到屏幕

1.安装elasticsearch，logstash，mysql，

2.同步mysql里的testdb数据库里的user表：
create table user(
 id int primary key auto_increment,
name varchar(800),
modify_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP)

3.下载mysql的jdbc驱动程序mysql-connector-j-8.0.32.jar到logstash安装目录：/usr/share/logstash/

4.新建配置文件 /usr/share/logstash/jdbc.conf

input {
 jdbc {
   jdbc_driver_library => "/usr/share/logstash/mysql-connector-j-8.0.32.jar*必须是绝对路径" 
   jdbc_driver_class => "com.mysql.jdbc.Driver"
   jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/testdb*数据库"
   jdbc_user => "mysql username"
   jdbc_password => "mysql password"
   jdbc_paging_enabled => true
   tracking_column => "unix_ts_in_secs"
   use_column_value => true
   tracking_column_type => "numeric"
   schedule => "*/5 * * * * *"
   statement => "*SQL语句：SELECT *, UNIX_TIMESTAMP(modify_time) AS unix_ts_in_secs FROM user WHERE (UNIX_TIMESTAMP(modify_time) > :sql_last_value AND modify_time < NOW()) ORDER BY modify_time ASC"
 }
}

filter {
 mutate {
   copy => { "id" => "[@metadata][_id]"}
   remove_field => ["id", "@version", "unix_ts_in_secs"]
 }
}

output {
 stdout { codec =>  "rubydebug"}
}
5.启动logstash
bin/logstash -f jdbc.conf

二、同步mysql数据到elasticsearch

1.修改配置文件jdbc.conf的output部分

output {
 elasticsearch {
   index => "user_idx"
   ilm_enabled => false
   ssl => false
   user => "elastic"
   password => "*elastic用户的密码"
 }
}

2.启动logstash

bin/logstash -f jdbc.conf

3.查看elasticsearch数据,打开sql客户端命令窗口

java -jar /elasticsearch安装目录/bin/elasticsearch-sql-cli-8.*.jar http://127.0.0.1:9200

sql>describe user_idx;

sql>select * from user_idx;

参考文献：

Ingest data from a relational database into Elasticsearch Service | Elasticsearch Service Documentation | Elastic