Performing Core Operations
-----------------------------------------------------------
date filter
解析字段中的日期,用作事件的Logstash时间戳。
filter {
date {
match => [ "logdate", "MMM dd yyyy HH:mm:ss" ]
}
}
-----------------------------------------------------------
drop filter
删除时间。通常与条件组合使用。
filter {
if [loglevel] == "debug" {
drop { }
}
}
-----------------------------------------------------------
fingerprint filter
通过应用一致的散列来打印字段。
filter {
fingerprint {
source => ["IP", "@timestamp", "message"]
method => "SHA1"
key => "0123"
target => "[@metadata][generated_id]"
}
}
-----------------------------------------------------------
mutate filter
在字段上执行一般改变。可以重命名、删除、替换和修改事件中的字段。
filter {
mutate {
rename => { "HOSTORIP" => "client_ip" }
}
}
filter {
mutate {
strip => ["field1", "field2"]
}
}
-----------------------------------------------------------
ruby filter
执行ruby代码
filter {
ruby {
code => "event.cancel if rand <= 0.90"
}
}
----------------------------------------------------------------------------------------------------------------------
Deserializing Data
数据反序列化插入到Logstash event
avro codec
csv filter
fluent codec
json codec
protobuf codec
xml filter
----------------------------------------------------------------------------------------------------------------------
Extracting Fields and Wrangling Data
提取字段,将非结构化数据解析为字段
-----------------------------------------------------------
dissect filter
使用分隔符将非结构化事件数据提取到字段中。剖析过滤器不使用正则表达式,并且非常快。然而,如果数据的结构因行而异,则grok过滤器更合适。
Exsample:
日志格式
2018-12-26 02:00:38,220 [Test-ELK] INFO id=54, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=68, c2=a, c3=f, c4=8jaufH
logstash命令:
./logstash -e 'input { beats { port => 5044 } } filter { dissect { mapping => { "message" => "%{ts} %{+ts},%{+ts} [%{logname}] %{loglevel} id=%{id}, myname=%{hostname}, myaddr=%{ip}, c1=%{c1}, c2=%{c2}, c3=%{c3}, c4=%{c4}" } } } output { stdout { } }'
结果:
{
"prospector" => {
"type" => "log"
},
"c4" => "8jaufH",
"ip" => "192.168.4.17",
"ts" => "2018-12-26 02:00:38,220",
"c1" => "68",
"@version" => "1",
"@timestamp" => 2018-12-26T07:00:40.425Z,
"source" => "/root/test_elk/test_elk.log",
"offset" => 305858,
"id" => "54",
"c2" => "a",
"logname" => "Test-ELK",
"host" => {
"name" => "dbwtest03bc.daodao.com",
"id" => "6787d9310dd84654ab8871f64df6f6d7",
"architecture" => "x86_64",
"os" => {
"platform" => "centos",
"codename" => "Core",
"family" => "redhat",
"version" => "7 (Core)"
},
"containerized" => true
},
"beat" => {
"name" => "dbwtest03bc.daodao.com",
"version" => "6.5.4",
"hostname" => "dbwtest03bc.daodao.com"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"message" => "2018-12-26 02:00:38,220 [Test-ELK] INFO id=54, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=68, c2=a, c3=f, c4=8jaufH",
"hostname" => "dbwtest03bc.daodao.com",
"c3" => "f",
"loglevel" => "INFO",
"input" => {
"type" => "log"
}
}
-----------------------------------------------------------
kv filter
解析键值对
Example:
日志
2018-12-26 02:08:08,946 [Test-ELK] INFO id=99, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=97, c2=c, c3=t, c4=mJPsIQ
logstash命令:
./logstash -e 'input { beats { port => 5044 } } filter { kv { } } output { stdout { } }'
输出
{
"beat" => {
"version" => "6.5.4",
"hostname" => "dbwtest03bc.daodao.com",
"name" => "dbwtest03bc.daodao.com"
},
"c1" => "97,",
"message" => "2018-12-26 02:08:08,946 [Test-ELK] INFO id=99, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=97, c2=c, c3=t, c4=mJPsIQ",
"@timestamp" => 2018-12-26T07:08:11.513Z,
"input" => {
"type" => "log"
},
"id" => "99,",
"myaddr" => "192.168.4.17,",
"host" => {
"architecture" => "x86_64",
"os" => {
"version" => "7 (Core)",
"family" => "redhat",
"codename" => "Core",
"platform" => "centos"
},
"containerized" => true,
"id" => "6787d9310dd84654ab8871f64df6f6d7",
"name" => "dbwtest03bc.daodao.com"
},
"c4" => "mJPsIQ",
"offset" => 311614,
"c2" => "c,",
"prospector" => {
"type" => "log"
},
"source" => "/root/test_elk/test_elk.log",
"myname" => "dbwtest03bc.daodao.com,",
"@version" => "1",
"c3" => "t,",
"tags" => [
[0] "beats_input_codec_plain_applied"
]
}
-----------------------------------------------------------
grok filter
将非结构化事件数据解析为字段。这个工具非常适合于syslog日志、Apache和其他web服务器日志、MySQL日志,以及一般情况下,任何为人类编写的日志格式,而不是计算机使用。Grok通过将文本模式组合成与日志匹配的内容来工作。
日志
2018-12-27 05:17:17,829 [Test-ELK] INFO id=19, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=10, c2=c, c3=f, c4=V6DSLg
命令
./logstash -e 'input { beats { port => 5044 } } filter { grok { match => { "message" => "%{DATESTAMP:time}%{SPACE}\[%{DATA:title}\]%{SPACE}%{LOGLEVEL:level}%{SPACE}%{GREEDYDATA:log_message}" } } } output { stdout { } }'
输出
{
"time" => "18-12-27 05:20:18,117",
"host" => {
"architecture" => "x86_64",
"name" => "dbwtest03bc.daodao.com",
"id" => "6787d9310dd84654ab8871f64df6f6d7",
"containerized" => true,
"os" => {
"platform" => "centos",
"codename" => "Core",
"version" => "7 (Core)",
"family" => "redhat"
}
},
"prospector" => {
"type" => "log"
},
"input" => {
"type" => "log"
},
"title" => "Test-ELK",
"message" => "2018-12-27 05:20:18,117 [Test-ELK] INFO id=37, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=37, c2=b, c3=f, c4=SrIzYL",
"offset" => 351361,
"@timestamp" => 2018-12-27T10:20:20.233Z,
"beat" => {
"version" => "6.5.4",
"name" => "dbwtest03bc.daodao.com",
"hostname" => "dbwtest03bc.daodao.com"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"level" => "INFO",
"@version" => "1",
"source" => "/root/test_elk/test_elk.log",
"log_message" => "id=37, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=37, c2=b, c3=f, c4=SrIzYL"
}
----------------------------------------------------------------------------------------------------------------------
Enriching Data with Lookups
dns filter
执行标准或反向DNS查找。
elasticsearch filter
将Elasticsearch中以前的日志事件中的字段复制到当前事件。
下面的配置显示了如何使用这个过滤器的完整示例。每当Logstash收到“end”事件时,它就使用这个Elasticsearch过滤器根据一些操作标识符查找匹配的“start”事件。然后,它将@timestamp字段从“start”事件复制到“end”事件的新字段中。最后,使用日期过滤器和ruby过滤器的组合,示例中的代码计算两个事件之间的时间持续时间(以小时为单位)。
geoip filter
添加关于IP地址位置的地理信息。
在应用geoip过滤器之后,事件将用geoip字段进行充实。
jdbc_static filter
使用从远程数据库预加载的数据丰富事件。
从远程数据库获取数据,将其缓存在本地数据库中,并使用查找来用缓存在本地数据库中的数据丰富事件。
jdbc_streaming filter
用数据库数据丰富事件。
translate filter
基于散列或文件中指定的替换值替换字段内容。目前支持这些文件类型:YAML、JSON和CSV。
useragent filter
将用户代理字符串解析为字段。