版权声明：版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/qq_37960324/article/details/82805277
ELK企业应用-elk监控nginx（二）logstash配置
1、编写正则表达式
根据日志结构，编写正则表达式，存储在/etc/logstash/patterns/nginx，内容如下：
URIPARM1 [A-Za-z0-9$.+!*'|(){},~@#%&/=:;^\\_<>`?\-\[\]]*
URIPATH1 (?:/[\\A-Za-z0-9$.+!*'(){},~:;=@#% \[\]_<>^\-&?]*)+
HOSTNAME1 \b(?:[0-9A-Za-z_\-][0-9A-Za-z-_\-]{0,62})(?:\.(?:[0-9A-Za-z_\-][0-9A-Za-z-:\-_]{0,62}))*(\.?|\b)
STATUS ([0-9.]{0,3}[, ]{0,2})+
HOSTPORT1 (%{IPV4}:%{POSINT}[, ]{0,2})+
FORWORD (?:%{IPV4}[,]?[ ]?)+|%{WORD}
URI1 (%{URIPROTO}://)?(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?
NGINXACCESS %{IP:remote_ip} - (%{USERNAME:remote_user}|-) \[%{HTTPDATE:log_timestamp}\] %{HOSTNAME:http_host} %{WORD:method} \"%{URIPATH1:uri}\" \"%{URIPARM1:param}\" %{BASE10NUM:http_status} (?:%{BASE10NUM:bytes}|-) \"(?:%{URI1:http_referrer}|-)\" (%{BASE10NUM:upstream_status}|-) (?:%{HOSTPORT1:upstream_ip}|-) (%{BASE16FLOAT:upstream_response_time}|-) (%{BASE16FLOAT:request_time}|-) (?:%{QUOTEDSTRING:user_agent}|-) \"(%{WORD:x_forword_for}|-)\"2、配置Logstash理论
2.1.配置IP归属地
通过在logstash中配置geoip数据库解析IP归属地，这里使用了开源的IP数据源用来分析客户端IP归属地。
cd /etc/logstash
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
tar zxvf GeoLite2-City.tar.gz
mv GeoLite2-City_20180921 GeoLite2-City
chown -R logstash:root GeoLite2-City/
cd ~2.2.Logstash服务端配置。
geoip {
source => "remote_ip"
target => "geoip"
database => "/etc/logstash/GeoLite2-City/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
geoip过滤器从geoip中匹配ip字段，显示该ip的地理位置，参数如下：
source：ip来源字段
target：指定插入的logstash字断目标存储为geoip
database：geoip数据库的存放路径
add_field: 增加的字段，坐标经度
add_field: 增加的字段，坐标纬度
2.3.配置字段类型
Logstash字段默认存储为string类型，为了便于后期分析和统计，需要对有些字段做格式转换，配置如下：
filter {
......
mutate {
convert => [
"http_status" , "integer",
"bytes" , "integer",
"upstream_status" , "integer",
"upstream_response_time" , "float",
"request_time", "float",
"[geoip][coordinates]", "float"
]
remove_field => "message"
}
......
}mutate插件用于数据的修改、删除、类型转换。
convert-http_status：请求响应码转换为integer类型；
convert-bytes：响应大小转换为integer类型；
convert-upstream-status：代理后端响应码转换为integer类型；
convert-upstream_response_time：代理后端响应时间转换为integer类型；
convert-request_time：代理后端请求时间转换为integer类型；
convert-[geoip][coordinates]：坐标经度转换为integer类型；
convert-[geoip][latitude]：坐标纬度转换为integer类型；
remove_field： 移除message 的内容，因为数据已经过滤了一份，这里不必在用到该字段了；
2.4.更改日期格式
日期格式处理很实用，可以根据需要对timestamp进行转换。
date {
match => [ "timestamp","yyyy-MM-dd:HH:mm:ss Z"]
}2.5.filter完整配置
filter {
if [type] == "nginx-access" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [
"message", "%{NGINXACCESS}"
]
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/etc/logstash/GeoLite2-City/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][latitude]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [
"http_status" , "integer",
"bytes" , "integer",
"upstream_status" , "integer",
"upstream_response_time" , "float",
"request_time", "float",
"[geoip][coordinates]", "float",
"[geoip][latitude]", "float"
]
remove_field => "message"
}
date {
match => [ "timestamp","yyyy-MM-dd:HH:mm:ss Z"]
}
}
}3、logstash配置操作
3.1.日志采集端配置
cd /etc/logstash/conf.d/
vim /etc/logstash/conf.d/test.conf
##########################################
input{
file{
type => "nginx-a.access"
path => "/var/log/nginx/a.access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
file{
type => "nginx-b.access"
path => "/var/log/nginx/b.access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [type] == "nginx-*.access" {
grok {
patterns_dir => "/etc/logstash/patterns"
match => [
"message", "%{NGINXACCESS}"
]
}
geoip {
source => "remote_addr"
target => "geoip"
database => "/etc/logstash/GeoLite2-City/GeoLite2-City.mmdb"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][latitude]", "%{[geoip][latitude]}" ]
}
mutate {
convert => [
"http_status" , "integer",
"bytes" , "integer",
"upstream_status" , "integer",
"upstream_response_time" , "float",
"request_time", "float",
"[geoip][coordinates]", "float",
"[geoip][latitude]", "float"
]
remove_field => "message"
}
date {
match => [ "timestamp","yyyy-MM-dd:HH:mm:ss Z"]
}
}
}
output{
if [type] == "nginx-a.access"{
redis{
host => "10.0.0.132"
data_type => "list"
key => "nginx-a.access"
port => 6379
db => 5
password => "123456"
}
}
if [type] == "nginx-b.access"{
redis{
host => "10.0.0.132"
data_type => "list"
key => "nginx-b.access"
port => 6379
db => 5
password => "123456"
}
}
}3.2.日志发送端配置
cd /etc/logstash/conf.d/
vim /etc/logstash/conf.d/test.conf
#################################################
input {
redis {
data_type => "list"
host => "10.0.0.132"
db => "5"
port => "6379"
key => "nginx-a.access"
password => "123456"
}
redis {
data_type => "list"
host => "10.0.0.132"
db => "5"
port => "6379"
key => "nginx-b.access"
password => "123456"
}
}
output {
if [type] == "nginx-a.access"{
elasticsearch {
hosts => ["10.0.0.130:9200"]
index => "a.nginx-%{+YYYY.MM.dd}"
}
}
if [type] == "nginx-b.access"{
elasticsearch {
hosts => ["10.0.0.130:9200"]
index => "b.nginx-%{+YYYY.MM.dd}"
}
}
}4、kibana页面配置
过一会后，kibana上出现配置的索引“a.nginx-2018.09.21”“b.nginx-2018.09.21”
创建索引，查看内容
监控正常
至此，elk监控nginx配置完毕。