win搭建ELK7.10.1分析Nginx日志绘制图表

前言

ELK分析日志非常强大，解析Nginx日志可以获得很多非常有价值的信息。

搭建

■ 下载

https://elasticsearch.cn/download/
中文社区下载对应安装包，速度更快。

■ 一些特殊配置

ES需要安装elasticsearch-head，然后

elasticsearch.yml添加配置处理跨域。

http.cors.enabled: true
http.cors.allow-origin: "*"

kibana.yml添加配置修改为中文

i18n.locale: "zh-CN"

kibana.yml添加配置修改地图为高德中文

map.tilemap.url: 'https://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

■ 修改Nginx日志格式

修改nginx.conf配置Nginx日志为json格式，并配置所需字段。

log_format main_json '{"domain":"$server_name",'
    '"remote_addr":"$remote_addr",'
    '"time_local":"$time_iso8601",'
    '"request":"$request",'
    #'"request_body":"$request_body",'
    '"status":$status,'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"http_referer":"$http_referer",'
    '"upstream_response_time":"$upstream_response_time",'
    '"request_time":"$request_time",'
    '"http_user_agent":"$http_user_agent",'
    '"upstream_addr":"$upstream_addr",'
    '"upstream_status":"$upstream_status"}';
  
	#用来指定日志文件的路径以及使用何种日志格式记录日志
	access_log  logs/access.log  main_json;

■ 配置logstash.conf

① 配置如下

input{
    
    
	file{
    
    
		path => ["路径/nginx-1.18.0/logs/access*.log"]	
		type => "nginx_access"
		start_position => "beginning"
	}
	file{
    
    
		path => ["路径/nginx-1.18.0/logs/error*.log"]	
		type => "nginx_error"
		start_position => "beginning"
	}
}
filter {
    
    
	if [type] == "nginx_access" {
    
    
		mutate {
    
    
			gsub => ["message", "\\x", "\\\x"]
		}
		json {
    
    
			source => "message"
			remove_field => "message"
			remove_field => "@timestamp"
		}
		date {
    
    
			match => ["time_local", "ISO8601"]
			target => "@timestamp"
		}
		grok {
    
    
			match => {
    
     "request" => "%{WORD:method} (?<url>.* )" }
		}
		mutate {
    
    
			remove_field => ["host", "event", "input", "offset", "prospector", "source", "tags", "beat"]
			rename => {
    
    "http_user_agent" => "agent"}
			rename => {
    
    "upstream_response_time" => "response_time"}
			convert => ["request_time", "float"]
			split => {
    
    "x_forwarded_for" => ","}
			split => {
    
    "response_time" => ","}
		}
		geoip {
    
    
            source => "remote_addr"
			target => "geoip"
			add_field => [ "[coordinates]", "%{[geoip][latitude]},%{[geoip][longitude]}" ]
        }
	}
	if [type] == "nginx_error" {
    
    
        mutate {
    
    
            remove_field => ["@timestamp"]
        }
        grok {
    
    
            match => {
    
    "message" => "(?<datetime>%{YEAR}[./-]%{MONTHNUM}[./-]%{MONTHDAY}[- ]%{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER}: %{GREEDYDATA:errormessage}(?:, client: (?<real_ip>%{IP}|%{HOSTNAME}))(?:, server: %{IPORHOST:domain}?)(?:, request: %{QS:request})?(?:, upstream: (?<upstream>\"%{URI}\"|%{QS}))?(?:, host: %{QS:request_host})?(?:, referrer: \"%{URI:referrer}\")?"}
        }
        date {
    
    
            match => ["datetime", "yyyy/MM/dd HH:mm:ss"]
            target => "@timestamp"
        }
    }
} 
output{
    
    
	elasticsearch {
    
    
	#这里记得写的是hosts，因为可能会是集群的 elasticsearch，但是配置的之后只要写上主节点就行了	
        hosts => "localhost:9200"
	#规定存储日志的索引规则，动态配置，方便管理
        index => "nginx-%{type}=%{+YYYY.MM.dd}"
		template => "路径/ELK/logstash-7.10.1/config/template/nginx-logstash.json"
		template_name => "nginx-logstash"
		template_overwrite => true
    }
	stdout{
    
    
		codec => rubydebug
	}
}

② 注意要点

即使是win下，路径也应使用/。
request_time转换为浮点型，便于计算平均值，最大值等。
定义的index索引规则必须和自己数据相匹配，自定义名称就需要自定义模板。由于需要绘制地图所以配置了模板以识别geo_point。
参考文章ELK7.10.1绘制热力地图Kibana无法识别geo_point