Personal blog navigation page (click on the right link to open a personal blog): Daniel take you on technology stack
The second week, and today we are speaking about Logstash use.
First, we open the ES, a new index using the postman user, using PUT way to localhost in the postman in: 9200 / user sends a request. Return information see:
{
"acknowledged": true,
"shards_acknowledged": true,
"index": "user"
}Description indexing is successful, if not established successfully, probably because ES does not open automatically create an index, modify config / elasticsearch.yml in the configuration, the action.auto_create_index set to true. Re-run at the above operations.
This time we look at user data in the index, used postman get in the way, sending localhost: 9200 / user / _search q = * & pretty view all data in the user index?. It can be seen in the user index and no data.
Our table structure to read as follows:
CREATE TABLE `user` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`account` varchar(40) DEFAULT NULL,
`password` varchar(40) DEFAULT NULL,
`name` varchar(20) DEFAULT NULL,
`sex` varchar(15) DEFAULT NULL,
`head_src` varchar(40) DEFAULT NULL,
`question` int(11) DEFAULT NULL,
`answer` varchar(100) DEFAULT NULL,
`signature` varchar(40) DEFAULT NULL,
`grade` int(11) DEFAULT NULL,
`code` varchar(100) NOT NULL,
`state` tinyint(1) unsigned zerofill NOT NULL,
`permission` tinyint(1) DEFAULT NULL,
`date` date DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `account` (`account`)
) ENGINE=InnoDB AUTO_INCREMENT=13 DEFAULT CHARSET=utf8 STATS_PERSISTENT=1;(The university system's own table, notes are not technical, but off ah)
Next, we enter Logstash folder, create a folder ConnectionJar, into the jar package mysql connection in this folder, you can go to the next Maven repository, where I hold the link.
Then we open the file in the bin Logstash the root folder, create conf folder, this folder, we create mysql.conf file. Open the conf file, set the file encoded as ANSI (very important !! important !! important !!).
The contents of the configuration file could look like:
input {
# 标准输入形式，暂时用不到
stdin {
}
# JDBC设置
jdbc {
# 数据库连接
jdbc_connection_string => "jdbc:mysql://localhost:3306/retina"
jdbc_user => "root"
jdbc_password => "123456"
# Jar包的位置
jdbc_driver_library => "D:\elasticSearch\logstash-7.6.0\ConnectionJar\mysql-connector-java-5.1.40-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
# 是否使用列数据
use_column_value => true
# 追踪的列，与use_column_value必须同时使用
tracking_column => id
# 是否开启记录最后一次运行数据
record_last_run => true
# 记录数据的保存位置
last_run_metadata_path=>"D:\elasticSearch\logstash-7.6.0\bin\conf\station_parameter.txt"
# 是否开启分页
jdbc_paging_enabled => "true"
# 分页数据大小
jdbc_page_size => "50000"
# SQL语句，也可以使用以下的文件形式
# statement_filepath => "D:\elasticSearch\logstash-7.6.0\bin\conf\information.sql"
statement => "select * from user where id > :sql_last_value "
# 定时执行的时间，这个在我的一篇博客里提到https://my.oschina.net/u/4109273/blog/3042086
schedule => "* * * * *"
# 索引的名称
type => "user"
}
}
# 拦截器设置，本篇中没有用到，就不解释了
filter {}
# 输出配置
output {
elasticsearch {
# ES的地址
hosts => "localhost:9200"
# 使用的索引
index => "user"
# 文档使用的ID列
document_id => "%{id}"
}
}but! You can not use the above configuration, will complain, because the file does not appear in Chinese, all we want to delete all the comments!
but! You can not use the above configuration, will complain, because the file does not appear in Chinese, all we want to delete all the comments!
but! You can not use the above configuration, will complain, because the file does not appear in Chinese, all we want to delete all the comments! (I write configuration information available at the end)
Then we open the file in bin cmd, run. \ Logstash -f. \ Conf \ mysql.conf command. See this screenshot, it shows the data processing Logstash already in progress.
Then we look at the data in the user index, we can see that the data has been written, and then compare the data user table, you can see, all the user data has been written to the ES. So far, the foundation has been using logstash it.
Configuration information:
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/retina"
jdbc_user => "root"
jdbc_password => "123456"
jdbc_driver_library => "D:\elasticSearch\logstash-7.6.0\JDBC\mysql-connector-java-5.1.40-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
use_column_value => true
tracking_column => id
record_last_run => true
last_run_metadata_path=>"D:\elasticSearch\logstash-7.6.0\bin\conf\station_parameter.txt"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
statement => "select * from user where id > :sql_last_value "
schedule => "* * * * *"
type => "user"
}
}
filter {}
output {
elasticsearch {
hosts => "localhost:9200"
index => "user"
document_id => "%{id}"
}
}Attached Java / C / C ++ / machine learning / Algorithms and Data Structures / front-end / Android / Python / programmer reading / single books books Daquan:
(Click on the right to open there in the dry personal blog): Technical dry Flowering
===== >> ① [Java Daniel take you on the road to advanced] << ====
===== >> ② [+ acm algorithm data structure Daniel take you on the road to advanced] << ===
===== >> ③ [database Daniel take you on the road to advanced] << == ===
===== >> ④ [Daniel Web front-end to take you on the road to advanced] << ====
===== >> ⑤ [machine learning python and Daniel take you entry to the Advanced Road] << ====
===== >> ⑥ [architect Daniel take you on the road to advanced] << =====
===== >> ⑦ [C ++ Daniel advanced to take you on the road] << ====
===== >> ⑧ [ios Daniel take you on the road to advanced] << ====
=====> > ⑨ [Web security Daniel take you on the road to advanced] << =====
===== >> ⑩ [Linux operating system and Daniel take you on the road to advanced] << = ====There is no unearned fruits, hope you young friends, friends want to learn techniques, overcoming all obstacles in the way of the road determined to tie into technology, understand the book, and then knock on the code, understand the principle, and go practice, will It will bring you life, your job, your future a dream.
