Overview
Detailed explanation of the use of the log system ELK (1) – How to use the
log system ELK detailed explanation (two) – Logstash installation and use of the
log system Detailed Explanation of ELK Use (5) – Supplement
Before we start, let's talk about the use and familiarity roadmap of Logstash. When you contact ELK, you can't be quick, you can try to install it part by part to become familiar with it, and then assemble it one by one to see the effect.
Logstash is very independent, and the familiar route can be followed as follows:
1. The most basic is to receive console input, and then parse the output to the console.
2. Read in from the file, parse the output to the console.
3. Read in from the file, parse the output to elasticsearch.
4. Joints that need to be opened up in practical applications.
Next, let's take a look at Logstash. The version used in this series (Logstash5.3.0) download address is: https://www.elastic.co/cn/downloads
Install
1. Install JDK 1.8.0_65
2. Download logstash5.3.0
3. Unzip logstash
Console -> Console
1. Create a new file std_std.conf in the .../logstash-5.3.0/bin/ directory
2. Enter the following content:
input {
stdin{
}
}
output {
stdout{
}
}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
3. Execute the command in the bin directory: ./logstash -f std_std.conf
4. Wait for a while and see the following information, indicating that the startup is complete:
5. Enter hello logstash!!!! , logash will write this information back to the console:
File-->Console
The file used here is the access log file output by tomcat, which is very common. The content is as follows:
111.206.36.140 - - [10/Aug/2016:23:16:29 +0800] "GET /nggirl-web/web/admin/work/special/listSelectedWork/v1.4.0?workIds=780 HTTP/1.1" 200 78 "http://www.baidu.com/s?wd=www" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"
111.206.36.140 - - [10/Aug/2016:23:16:29 +0800] "GET /nggirl-web/web/admin/work/special/listSelectedWork/v1.4.0?workIds=780 HTTP/1.1" 200 78 "http://www.baidu.com/s?wd=www" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)"- 1
- 2
1. Create a new file_std.conf file in the bin directory;
2. The contents are as follows:
input{
file{
path =>"/Develop/Tools/apache-tomcat-8.0.30/webapps/nggirllog/access.log"
start_position=>"beginning"
}
}
filter{
grok{
match=>{
"message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""
}
remove_field=>"message"
}
date{
match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
}
}
output{
stdout{
codec=>rubydebug
}
}
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
3. Then run ./logstash -f file_std.conf, you will see that the original content in the file is displayed on the command line one by one.
This configuration file is relatively complicated, and when we generally use elk, the configuration file of logstash is basically similar to this, with similar differences. Here, we will explain the key parts in detail. For more in-depth content and usage, you can go to the elastic official website or the best practice page of logstash, the website is: http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html .
input/file/path: This specifies which log file to scan. If you want to scan multiple files, you can use * this path wildcard; or use multiple log paths to provide in the form of an array (path=>["outer-access.log", "access.log"]); or directly give a directory, logstash will scan all files and listen for new files.
filter/grok/match/message: Both DATA and HTTPDATE are built-in regular expressions in grok syntax. DATA matches any character, and HTTPDATE matches joda-type date format characters. In the above example "\[" matches "[".
filter/grok/match/date: is an explanation of the HTTPDATE date format. Joda can support many complex date formats, which need to be specified here to match correctly.
remove_field=>”message”：用处是去掉原有的整个日志字符串，仅保留filter解析后的信息。你可以试着去掉这一句就明白他的用处了。
解析成功后会看到控制台中类似如下的内容：
文件到elasticsearch
1.在bin目录新建file_es.conf文件
2.录入如下内容，和上一个例子的区别仅在于out部分：
input{
file{
path =>"/Develop/Tools/apache-tomcat-8.0.30/webapps/nggirllog/access*.log"
start_position=>"beginning"
}
}
filter{
grok{
match=>{
"message"=>"%{DATA:clientIp} - - \[%{HTTPDATE:accessTime}\] \"%{DATA:method} %{DATA:requestPath} %{DATA:httpversion}\" %{DATA:retcode} %{DATA:size} \"%{DATA:fromHtml}\" \"%{DATA:useragent}\""
}
remove_field=>["message"]
}
date{
match=>["accessTime","dd/MMM/yyyy:HH:mm:ss Z"]
}
}
output {
elasticsearch {
hosts => "127.0.0.1"
}
stdout { codec => rubydebug}
}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
3.执行./logstash -f file_es.conf
由于我们还没有开始部署elasticsearch，暂时不贴出效果，效果图将在下一个博客里面看到。
多行日志的处理
通过上面3步的熟悉，我们大概清楚了logstash的工作过程：
1.input读取指定文件里面的文本行，这里是一行一行读取的；
2.然后filter对读入的每一行进行解析，拆分成一组一组的key-value值；
3.out将解析后的结果输出写入到指定的系统。
但是，我们知道异常日志是一个多行文本，我们需要把多行信息输出到一行里面去处理，那么怎么办呢？
如果仍然按照单行文本的类似处理方式的话，我们需要在input部分添加配置项，如下：
input {
stdin {
codec => multiline {
pattern => "^\["
negate => true
what => "previous"
}
}
}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
这个配置是对每一个以”[“开头的行认作一个数据行的开始，后续不是以”[“开头的行都会认为是同一个数据的内容，当再次遇到”[“时结束这一个数据内容的读取，开始下一行。
这里有一个问题是，最后一个异常日志不能输出，因为logstash一直在等待下一个”[“的出现，但是一直没有出现，那么就不会输出了。
我们采用的处理多行文本的方式是自己实现了一个LogAppender，直接由程序日志类库以json串的形式输出到redis中，然后再由logstash读取。架构方式上属于第一篇中写的第二种架构方式。
以下是一些关键部分的代码片段:
Supplement: If you need to block unnecessary logs, you can refer to here: http://www.tuicool.com/articles/Ubeiaea
#########################
Reprinted from: https://blog.csdn.net/buqutianya/article/details/72019264?utm_source=itdadao&ut