Table of contents
2.1, mysql database is available
2.2. Download the dependency package mysql-connector-java-8.0.30.jar
Expansion: How to download the dependencies of the maven official website
2.3. Create script file find.sql
2.4. Create configuration file mysql.conf
3. Start Logstash to start mysql table data backup
Problem phenomenon:
Preface (optional)
Recently, I was revisiting Elasticsearch. It seems that the official website has been released to version 8.4.3. I think I used 5.6.8 when I was a beginner, and the version has been updated a lot, which means a big change.
The windows environment is based on the Logstash tool of Elasticsearch8.4.0 to realize the backup and synchronization of Mysql database table data!
Solution:
1. Download the Logstash tool
Elastic official website download:
Past Releases of Elastic Stack Software | Elastic
Note that the version number should be the same as the version number of Elasticsearch, so here I downloaded 8.4.0, click WINDOWS to start the download of the zip package:
After the download is complete, extract it to a custom location, such as (D:\logstash-8.4.0).
2. Configuration
2.1, mysql database is available
First of all, make sure that the mysql service has been started and the database can be connected (here I use the student table in the mysqltest database of my mysq service for practice):
2.2. Download the dependency package mysql-connector-java-8.0.30.jar
Create a new folder mysql in the decompressed logstash directory:
You can find and download the dependency package on the official website of the maven warehouse. Here I downloaded version 8.0.30:
Put the downloaded dependency package into the newly created mysql directory above.
Expansion: How to download the dependencies of the maven official website
Let’s briefly mention how to download dependencies here. In fact, just recall how we usually introduce dependencies in Spring/Springboot projects.
Copy the red box in the above picture to the pom.xml file of the Spring/Springboot project, and it will usually be downloaded automatically, or you can manually click the Reimport button to start the download:
2.3. Create script file find.sql
In the mysql directory, create a new file find.sql, and write the query sql used for data backup , such as :
-- student 为需要导入的表
select * from student2.4. Create configuration file mysql.conf
In the mysql directory, create a new file mysql.conf and write the following configuration :
input {
stdin {
}
jdbc {
# mysql 数据库链接,jdbc版本比较大的要加上？后面那串字符
jdbc_connection_string => "jdbc:mysql://localhost:3306/数据库名?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true"
# 用户名和密码
jdbc_user => "用户名"
jdbc_password => "密码"
# 驱动
jdbc_driver_library => "mysql-connector-java.jar包所在路径"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
# 执行的sql 就是上一步创建的sql文件的绝对路径+文件名字
statement_filepath => "find.sql文件所在路径"
# 也可以写一个要执行sql语句，替代statement_filepath的方式
#statement => "select * from tb_article"
# 设置监听间隔 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
schedule => "* * * * *"
# 索引类型
type => "student"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
# ES的IP地址及端口
hosts => ["localhost:9200"]
# 索引名称，elasticsearch叫做索引,相当于es的数据库
index => "自定义索引名"
# 自增ID id必须是待查询的数据表的序列字段
document_id => "%{表的主键字段名}"
}
stdout {
# JSON格式输出
codec => json_lines
}
}Here is also my own configuration for your reference :
input {
stdin {
}
jdbc {
# mysql 数据库链接,center为数据库名,jdbc版本比较大的要加上？后面那串字符
jdbc_connection_string => "jdbc:mysql://localhost:3306/mysql_test?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true"
# 用户名和密码
jdbc_user => "root"
jdbc_password => "123456"
# 驱动
jdbc_driver_library => "D:\logstash-8.4.0\mysql\mysql-connector-java-8.0.30.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
# 执行的sql 就是上一步创建的sql文件的绝对路径+文件名字
statement_filepath => "D:\logstash-8.4.0\mysql\find.sql"
# 也可以写一个要执行sql语句，替代statement_filepath的方式
#statement => "select * from tb_article"
# 设置监听间隔 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为每分钟都更新
schedule => "* * * * *"
# 索引类型
type => "student"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
# ES的IP地址及端口
hosts => ["localhost:9200"]
# 索引名称，elasticsearch叫做索引,相当于es的数据库
index => "mysql"
# 自增ID id必须是待查询的数据表的序列字段
document_id => "%{id}"
}
stdout {
# JSON格式输出
codec => json_lines
}
}This is what it looks like when it's done:
3. Start Logstash to start mysql table data backup
Open the command line window, go to the bin directory of logstash after the zip package is decompressed, and enter:
//logstash -f mysql.conf文件路径,如：
logstash -f D:\logstash-8.4.0\mysql\mysql.conf
Backup succeeded:
Use the Head tool to view:
