Foreword:
I originally wanted to give a brief introduction, but there is more to write. Thank you for your blogs and ideas. This blog also uses the feature of ubiquitous links. I hope it will help you.
Why have I started blogging frequently recently (I only feel like I'm just getting started) (there are other things that need to go out, in order to better bring convenience to the people behind), I wrote all the things I have sorted out, and everyone is happy to help them, no Help, please don’t complain, it’s okay to complain, I try not to be boring and sulky because of the outside world
introduce
Logstash: A data processing engine that can process tens of thousands of logs per second; it supports dynamically collecting data from various data sources, filtering, analyzing, enriching, and unifying the data, and then storing it in ES
The official website accepts: both pictures and texts, very good
Log collection and processing framework - [Logstash] detailed use
logstash做的事情分三个阶段依次执行：输入——》处理filter（不是必须）——》输出
使用管道方式进行日志的搜集处理和输出：有点类似*NIX系统的管道命令 xxx | ccc | ddd，xxx执行完了会执行ccc，然后执行ddd
As shown in the figure:
Install
1. Installation: You can download tar.gz, just unzip it.
2. Command installation: wget https://artifacts.elastic.co/downloads/logstash/logstash-5.2.1.tar.gz
Detailed explanation:
As mentioned above, logstash supports dynamic collection of data from various data sources
1. Dynamic means that the specified data can be collected in real time after relevant configuration
2. Various data sources, how to do this? There is an input in the configuration used by logstash for collection . In order to facilitate a systematic understanding, let's go to the map.
In order to facilitate management, we can write his configuration into a separate file, which is the configuration file (files ending in .conf, used to manage input, filter and output-related configuration):
input accepts data input, score input way (plug-in support is required)
to output the output file to the specified place, the plug-in is required to support
filters and codecs: operations such as filtering, analyzing, enriching, and unifying the data
The macro configuration file content format is as follows:
# 输入
input {
...
}
# 过滤器
filter {
...
}
# 输出
output {
...
}configuration file
In fact, the above has already started to talk about the configuration file;
at present I use a relatively simple one, just use a grok, so the other needs to be studied. I
just found a blog by [Guan Xiaoxi] on the Internet, and I feel that it is a very good introductory tutorial:
Combined with the big guy's blog, paste a configuration file out:
input {
# 从文件读取日志信息
file {
path => "/var/log/error.log"
type => "error"//type是给结果增加一个type属性,值为"error"的条目
start_position => "beginning"//从开始位置开始读取
# 使用 multiline 插件，传说中的多行合并
codec => multiline {
# 通过正则表达式匹配，具体配置根据自身实际情况而定
pattern => "^\d"
negate => true
what => "previous"
}
}
}
#可配置多种处理规则，他是有顺序，所以通用的配置写下面
# filter {
# grok {
# match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
# }
output {
# 输出到 elasticsearch
elasticsearch {
hosts => ["192.168.22.41:9200"]
index => "error-%{+YYYY.MM.dd}"//索引名称
}
}The above file can be configured with multiple:
file {
type => "tms_inbound.log"
path => "/JavaWeb/tms2.wltest.com/logs/tms_inbound.es.*.log"
codec => json {
charset => "UTF-8"
}
}
file {
type => "tms_outbound.log"
path => "/JavaWeb/tms2.wltest.com/logs/tms_outbound.es.*.log"
codec => json {
charset => "UTF-8"
}
} start up
Next, you can save the file as the testLogs.conf file, save it to the bin path of logstash, and enter the bin path of logstash to run:
bin/logstash -f testLogs.conf
//如果放到conf文件夹下
bin/logstash -f conf/testLogs.conf//路径要对应上，这里是conf/testLogs.confThis blog, I should have seen it before, it is also quite good, you can learn from it
After logstash, you can add:
-f: specify the configuration file, configure logstash according to the configuration file
-e: string, configuration, default "" stdin input, stdout output (input and output in the console), can be set through the command line
-l : output address, the default console output
-t: exit after testing whether the configuration file is correct
such as :
bin/logstash -e 'input { stdin { } } output { stdout {} }'-e accepts settings via command line
bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }'
bin/logstash -e 'input { stdin { } } output { elasticsearch { host => localhost } }'You can also start by writing a configuration file as mentioned above:
#写一个配置文件，放到bin目录下，启动
bin/logstash -f logstash-simple.conf
bin/logstash -f ./config/
bin/logstash -f etc/#加载 etc文件夹下所有 *.conf 的文本文件
#后台运行nohup bin/logstash -f etc/ &No matter how many configuration files are specified at startup, logstash will simply integrate all configuration files together and compile them into one file at startup. However, if there is no distinction, there may be repeated reading problems (a file is executed once , The b file is executed once: repeated reading, um, no problem), in this case, you can use the type (see the above example for usage) or tags to separate the data from different sources, and then at the time of output, filter Do grok regular When parsing, use if judgment to select different destinations or indexes
if "from_sys" in [tags]
#或者：
if [tags][0] == "achie_log"{
elasticsearch {
……
}
}
if [tags][0] == "exam_log"{
elasticsearch {
……
}
}Plugin installation
Logstash supports some plugins by default (with their own plugins), but if the plugins you want to use are not available in logstash itself, you need to install them:
First of all, in order to prevent the download from being too slow or unable to download, let's change the place first:
Enter the installation directory of logstash, modify the source item of the file named by Gemfile, and let the plugin download from China:
source " https://ruby.taobao.org/ "
Installation command: 1. Download the search keyword from https://github.com/logstash-plugins
in github to find the specified plug-in, download it, there is an installation introduction below the relevant homepage
2. Or the name of the plugin to be installed by bin/plugin install.
I usually copy the name from GitHub, copy it over, and manually enter it to worry about mistakes
This is OK. If there are few plug-ins when running the configuration file, logstash will give a prompt. Install the corresponding plug-ins according to the prompt. I believe everyone's strength.
Afterword : I just found a configuration summary
at 12:50:53 on April 21, 2018 , the commonly used configuration, you can refer to this with the help of this