Logstash plug-ins:
input plug-ins:
File: Read event stream from the specified file;
Use change monitor file FileWatch (Ruby Gem library).
.sincedb: Record the inode of each file being monitored, major number, minor nubmer, pos;
It is a simple example of log collection:
input {
file {
path => ["/var/log/messages"]
type => "system"
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
}
[ "/ Var / log / messages"] can contain multiple files [item1, item2, ...] start_position => "beginning" indicates the start of reading from the first row
udp: Message read from a network connection via the udp protocol, which is a mandatory parameter Port, for indicating their listening port, host address is specified by its own monitor
collectd: performance monitoring program, based on c language development, running in daemon mode, data can be collected in all aspects of system performance, and stores the result of the collection of down, can pass
Over the network plug-ins, sends its own data collected in this machine to another host
collectd in epel packet source, yum -y install epel-release; yum -y install collectd, collecctd profiles for /etc/collectd.conf
vim /etc/collectd.conf, will set up a Hostname under the name Global settings for the daemon: Hostname "node1"
Find LoadPlugin section, the LoadPlugin df uncommented, LoadPlugin network start
In <Plugin network> </ Plugin> The following further definitions section:
<Plugin network>
<Server "192.168.204.135" "25826">
</Server>
</Plugin>
192.168.204.135 represents the data to the host, the host listens on port 25826
service collectd start
192.168.204.135 installed lostash, following is an example of a UDP profile
input {
udp {
port => 25826
codec => collectd {}
type => "collectd"
}
}
output {
stdout {
codec => rubydebug
}
}
Information codec => collectd {} transmitted over the collectd do special coding
type => "collectd" type can be freely named
logstash -f /etc/logstash/conf.d/udp.conf --configtest logstash -f /etc/logstash/conf.d/udp.conf
This is can receive the information of the collectd
redis plugin:
从redis读取数据，支持redis channel和lists两种方式
filter插件：
用于在将event通过output发出之前对其实现某些处理功能
grok：用于分析并结构化文本数据；目前 是logstash中将非结构化日志数据转化为结构化的可查询数据的不二之选。
syslog, apache, nginx
模式定义位置：/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns
语法格式：
%{SYNTAX:SEMANTIC}
SYNTAX：预定义模式名称；
SEMANTIC：匹配到的文本的自定义标识符；
例如：1.1.1.1 GET /index.html 30 0.23
{ "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %
{NUMBER:duration}" }
vim groksample.conf 一个配置示例
input {
stdin {}
}
filter {
grok {
match => { "message" => "%{IP:clientip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %
{NUMBER:duration}" }
}
}
output {
stdout {
codec => rubydebug
}
}
logstash -f /etc/logstash/conf.d/groksample.conf --configtest
logstash -f /etc/logstash/conf.d/groksample.conf
输入1.1.1.1 GET /index.html 30 0.23， 得出结果
1.1.1.1 GET /index.html 30 0.23
{
"message" => "1.1.1.1 GET /index.html 30 0.231.1.1.1 GET /index.html 30 0.23",
"@version" => "1",
"@timestamp" => "2016-07-20T11:55:31.944Z",
"host" => "centos7",
"clientip" => "1.1.1.1",
"method" => "GET",
"request" => "/index.html",
"bytes" => "30",
"duration" => "0.231"
}
自定义grok的模式：grok的模式是基于正则表达式编写，其元字符与其它用到正则表达式的工具awk/sed/grep/pcre差别不大
自定义的机会一般不大
匹配apache log示例 vim apachesample.conf
input {
file {
path => ["/var/log/httpd/access_log"]
type => "apachelog"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
output {
stdout {
codec => rubydebug
}
}
nginx log的匹配方式：
将如下信息添加至 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.3.0/patterns/grok-patterns文
件的尾部
#Nginx log
NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - %{NOTSPACE:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%
{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %
{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}
yum -y install epel-release;yum -y install nginx;systemctl start nginx
vim nginxsample.conf
input {
file {
path => ["/var/log/nginx/access.log"]
type => "nginxlog"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
output {
stdout {
codec => rubydebug
}
}
logstash -f /etc/logstash/conf.d/nginxsample.conf
转载于:https://www.cnblogs.com/linuxboke/p/5689666.html