ELK7.4-Lostash Grok formatted log Nginx

Niche blog: http://xsboke.blog.51cto.com

                -------谢谢您的参考，如有疑问，欢迎交流

---

table of Contents

• demand
• surroundings
• GrokThe official introduction
• WebEnd arrangement
• Partially effective showcase

---

demand

使用filebeat收集nginx访问日志,output到logstash,
logstash使用grok filter plugin将获取到的nginx日志结构化
将结构化的nginx日志output到elasticsearch

surroundings

• Here are just and grokthe relevant configuration, other configurations refer to the article ELK7.4- Quick Start for data collection .

  web               172.16.100.251      nignx/filebeat/logstash 
  elasticsearch 172.16.100.252      elasticsearch/kibana

GrokThe official presentation

1. Parsing any text and structured.

2. GrokIt is a good way to unstructured log data analysis and query structured content.

3. This tool is ideal for sysloglogs apacheand other Webserver logs, mysqllogs, and any log format is commonly used human rather than computers.

4. Grokgrammar:%{SYNTAX:SEMANTIC}

   • SYNTAX: Matching syntax, which is the Groksyntax Attachment: Click here: Grok Patterns
   • SEMANTIC: Identifies the matched string is field.

5. The official recommended a web tool to verify Grok grammar , usage, will need to parse the string is written to the first text input box, Grok written expression written to the second input box, and then check Named Captures Onlyto .

   Example:

   • Nginx Log Format:$request_time|$host|$remote_addr|[$time_local]
   • Nginx Log:0.123|baidu.com|192.168.0.1|[18/Oct/2019:11:22:14 +0800]
     -
   • Tips, in order to avoid the "|" is escaped, use of "\" prohibit their escape.

webConfiguration

1. Nginx Log Format

# 这里的日志格式比较复杂,是为了更好的展示Grok
log_format access   '$request_time|$host|$remote_addr|$remote_user|[$time_local]|$request|$status|$upstream_status|$upstream_response_time|$upstream_addr|$body_bytes_sent|$request_body|$http_referer|$http_user_agent|$http_x_forwarded_for|$http_x_forwarded_path,$server_addr|$upstream_cache_status';

2. filebeat

vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/jpg_access.log
  tags: ["nginx_access"]

3. logstash

vim /etc/logstash/conf.d/nginx.conf

input {
    beats {
        port => 5044
    }
}

filter {
    if "nginx_access" in [tags] {
        grok {
            match => { "message" => "%{NUMBER:request_time}\|%{IPORHOST:host}\|%{IPORHOST:remote_addr}\|%{USERNAME:remote_user}\|\[%{HTTPDATE:time_local}\]\|%{NOTSPACE:request_method} %{NOTSPACE:request} (?:HTTP/%{NUMBER:http_version})\|%{NUMBER:status}\|%{NUMBER:upstream_status}\|%{NUMBER:upstream_response_time}\|%{NOTSPACE:upstream_addr}\|%{NUMBER:body_bytes_sent}\|%{NOTSPACE:request_body}\|%{NOTSPACE:http_referer}\|%{GREEDYDATA:http_user_agent}\|%{NOTSPACE:http_x_forwarded_for}\|%{NOTSPACE:http_x_forwarded_path}\|%{NOTSPACE:upstream_cache_status}" }
        }

        geoip {
            source => "http_x_forwarded_for" # 通过geoip库查询IP归属地
        }
    }
}

output {
    if "nginx_access" in [tags] {
        elasticsearch {
            hosts => ["172.16.100.252"]
            index => "nginx_access-%{+YYYY.MM.dd}"
        }
    }
}

effect

Then Kibanaadd the index above, Discoverwe will look at a multi-page customfields

So that is more conducive to data analysis later, and Discovercan be more intuitive filtering or view the data pages.