ELK 之Logstash(部署、使用和配置扩展插件)

Logstash是什么?

• 是一个数据采集,加工处理以及传输的工具
• 特点
  – 所有类型的数据集中处理
  – 不同模式和格式数据的正常化
  – 自定义日志格式的迅速扩展
  – 为自定义数据源轻松添加插件
• Logstash 安装
  – Logstash依赖java环境,需要安装java-1.8.0-openjdk
  –Logstash 没有默认的配置文件,需要手动配置
  –Logstash安装在/opt/logstash目录下
• Logstash工作结构
  {数据源} == >
  ~~ input { } = =>
  ~~~~filter { } = =>
  ~ ~~~~~output { }= =>
  ~ ~~~~~~~~~~~{ ES }

Logstash类型及条件判断

• Logstash 里面的类型
  – 布尔值类型: ssl_enable = > true
  – 字节类型: bytes = > “1MiB”
  – 字符串类型: name = > “xkops”
  – 数值类型: post = > 22
  – 数组: match = > [“datetime”,“UNIX”]
  – 哈希: options = > {k = > “v”, k2 = > “v2” }
  – 编码解码: codec = > “json”
  – 路径: file_path = > “/tmp/filename”
  – 注释: #

• Logstash 条件判断
  – 等于: ==
  – 不等于: !=
  – 小于 : <
  – 大于 : >
  – 小于等于: <=
  – 大于等于: >=
  – 匹配正则: =~
  –不匹配正则: !~
  – 包含: in
  – 不包含: not in
  – 与 : and
  – 或 : or
  – 非与: nand
  – 非或: xor
  – 复合表达式: ()
  – 取反复合: !()

  Logstash的安装配置

  服务器硬件要求:

  • 2CPU 4G内存 设置IP :192.168.1.31

[root@logstash ~]# yum -y install java-1.8.0-openjdk  logstash
[root@logstash ~]# java -version

openjdk version “1.8.0_161”
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

[root@logstash ~]#  /opt/logstash/bin/logstash  --version
logstash 2.3.4
[root@logstash ~]# /opt/logstash/bin/logstash-plugin  list   //查看插件
...
logstash-input-stdin    //标准输入插件
logstash-output-stdout    //标准输出插件
...
[root@logstash ~]# cd /etc/logstash
[root@logstash ~]# vim logstash.conf
input{
    
    
	stdin {
    
    }
}
filter {
    
    }
output{
    
    
	stdout {
    
    }
}
[root@logstash ~]# ls /opt/logstash
[root@logstash ~]# ls /opt/logstash/bin
[root@logstash ~]# /opt/logstash/bin/logstash -h   // 查看帮助信息
[root@logstash ~]# /opt/logstash/bin/logstash -f logstash
出现 main started 表示启动成功

备注：若不会写配置文件可以找帮助，插件文档的位置：

帮助文件地址点击链接

codec类插件

[root@logstash ~]# vim /etc/logstash/logstash.conf

input{
    
    
    stdin{
    
    
    codec => "json"        //输入设置为编码json
   }
}
filter{
    
    
}
output{
    
    
    stdout{
    
    
    codec => "rubydebug"        //输出设置为rubydebug
   }
}
[root@logstash ~]#  /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf 
Settings: Default pipeline workers: 2
Pipeline main started
{
    
    "a":1}
{
    
    
             "a" => 1,
      "@version" => "1",
    "@timestamp" => "2019-03-12T03:25:58.778Z",
          "host" => "logstash"
}

file模块插件

[root@logstash ~]# vim /etc/logstash/logstash.conf
input{
file {
path => [ “/tmp/a.log”, “/tmp/b.log” ]
sincedb_path => “/var/lib/logstash/sincedb” //记录读取文件的位置
start_position => “beginning” //配置第一次读取文件从什么地方开始
type => “testlog” //类型名称
}
}
filter{
}
output{
stdout{
codec => “rubydebug”
}
}
[root@logstash ~]# touch /tmp/a.log
[root@logstash ~]# touch /tmp/b.log
[root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf

• 另开一个终端：写入数据

[root@logstash ~]# echo a1 > /tmp/a.log
[root@logstash ~]# echo b1 > /var/tmp/b.log

• 回到之前的终端查看：
  [root@logstash ~]# /opt/logstash/bin/logstash -f /etc/logstash/logstash.conf
  Settings: Default pipeline workers: 2
  Pipeline main started
  {
  “message” => “a1”,
  “@version” => “1”,
  “@timestamp” => “2019-03-12T03:40:24.111Z”,
  “path” => “/tmp/a.log”,
  “host” => “logstash”,
  “type” => “testlog”
  }
  {
  “message” => “b1”,
  “@version” => “1”,
  “@timestamp” => “2019-03-12T03:40:49.167Z”,
  “path” => “/tmp/b.log”,
  “host” => “logstash”,
  “type” => “testlog”
  }

filter grok插件

grok插件：
解析各种非结构化的日志数据插件
grok使用正则表达式把飞结构化的数据结构化
在分组匹配，正则表达式需要根据具体数据结构编写
虽然编写困难，但适用性极广
解析Apache的日志
[root@es1 ~]# yum -y install httpd
[root@es1 ~]# systemctl restart httpd

• 浏览器访问网页，在/var/log/httpd/access_log有日志出现

[root@es5 ~]# cat /var/log/httpd/access_log
192.168.1.254 - - [12/Mar/2019:11:51:31 +0800] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
[root@logstash ~]#  vim /etc/logstash/logstash.conf
input{
    
    
    file {
    
    
      path           => [ "/tmp/a.log", "/tmp/b.log" ]
      sincedb_path   => "/var/lib/logstash/sincedb"
      start_position => "beginning"
      type           => "testlog"
   }
}
filter{
    
    
    grok{
    
    
       match => [ "message",  "(?<key>reg)" ]
    }
}
output{
    
    
    stdout{
    
    
    codec => "rubydebug"
   }
}
复制/var/log/httpd/access_log的日志到logstash下的/tmp/a.log
[root@logstash ~]# vim /tmp/a.log
192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] "GET / HTTP/1.1" 403 4897 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"
[root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf
//出现message的日志，但是没有解析是什么意思
Settings: Default pipeline workers: 2
Pipeline main started
{
    
    
       "message" => ".168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET / HTTP/1.1\" 403 4897 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
      "@version" => "1",
    "@timestamp" => "2018-09-15T10:26:51.335Z",
          "path" => "/tmp/a.log",
          "host" => "logstash",
          "type" => "testlog",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

– 若要解决没有解析的问题，同样的方法把日志复制到/tmp/a.log，logstash.conf配置文件里面修改grok

• 查找正则宏路径

[root@logstash ~]# cd  /opt/logstash/vendor/bundle/ \ 
jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/
[root@logstash ~]# vim grok-patterns  //查找COMBINEDAPACHELOG
COMBINEDAPACHELOG %{
    
    COMMONAPACHELOG} %{
    
    QS:referrer} %{
    
    QS:agent}
[root@logstash ~]#  vim /etc/logstash/logstash.conf
...
filter{
    
    
   grok{
    
    
        match => ["message", "%{COMBINEDAPACHELOG}"]
  }
}
...

• 解析出的结果

 [root@logstash ~]#  /opt/logstash/bin/logstash -f  /etc/logstash/logstash.conf  
Settings: Default pipeline workers: 2
Pipeline main started
{
    
    
        "message" => "192.168.1.254 - - [15/Sep/2018:18:25:46 +0800] \"GET /noindex/css/open-sans.css HTTP/1.1\" 200 5081 \"http://192.168.1.65/\" \"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\"",
       "@version" => "1",
     "@timestamp" => "2018-09-15T10:55:57.743Z",
           "path" => "/tmp/a.log",
ZZ           "host" => "logstash",
           "type" => "testlog",
       "clientip" => "192.168.1.254",
          "ident" => "-",
           "auth" => "-",
      "timestamp" => "15/Sep/2019:18:25:46 +0800",
           "verb" => "GET",
        "request" => "/noindex/css/open-sans.css",
    "httpversion" => "1.1",
       "response" => "200",
          "bytes" => "5081",
       "referrer" => "\"http://192.168.1.65/\"",
          "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0\""
}
...

步骤二： 安装Apache服务，用filebeat收集Apache服务器的日志，并存入elasticsearch

• 1）在之前安装了Apache的主机上面安装filebeat

[root@se5 ~]#  yum -y install filebeat
[root@se5 ~]#  vim/etc/filebeat/filebeat.yml  
paths:
    - /var/log/httpd/access_log   //日志的路径，短横线加空格代表yml格式
document_type: apachelog    //文档类型
elasticsearch:        //加上注释
hosts: ["localhost:9200"]                //加上注释
logstash:                    //去掉注释
hosts: ["192.168.1.57:5044"]     //去掉注释,logstash那台主机的ip
[root@se5 ~]# systemctl start filebeat
[root@logstash ~]#  vim /etc/logstash/logstash.conf
input{
    
    
        stdin{
    
     codec => "json" }
        beats{
    
    
            port => 5044
}
  file {
    
    
    path          => [ "/tmp/a.log", "/tmp/b.log" ]
   sincedb_path   => "/var/lib/logstash/sincedb"
   start_position => "beginning"
   type           => "testlog"
  }
filter{
    
    
if [type] == "apachelog"{
    
    
   grok{
    
    
        match => ["message", "%{COMBINEDAPACHELOG}"]
  }}
}
output{
    
    
      stdout{
    
     codec => "rubydebug" }
      if [type] == "filelog"{
    
    
      elasticsearch {
    
    
          hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
          index => "filelog"
          flush_size => 2000
          idle_flush_time => 10
      }}
}
 [root@logstash logstash]#  /opt/logstash/bin/logstash  \ 
-f  /etc/logstash/logstash.conf

• 打开另一终端查看5044是否成功启动

[root@logstash ~]#  netstat -antup | grep 5044
tcp6       0      0 :::5044                 :::*                    LISTEN      23776/java
[root@se5 ~]#  firefox 192.168.1.55   //ip为安装filebeat的那台机器
回到原来的终端，有数据

2）修改logstash.conf文件

[root@logstash logstash]# vim logstash.conf
...
output{
    
    
      stdout{
    
     codec => "rubydebug" }
      if [type] == "apachelog"{
    
    
      elasticsearch {
    
    
          hosts => ["192.168.1.51:9200", "192.168.1.52:9200"]
          index => "apachelog"
          flush_size => 2000
          idle_flush_time => 10
      }}
}

• 浏览器访问Elasticsearch，有apachelog，如图-20所示：

[student@room9pc01 ~]$ firefox http://192.168.1.55:9200/_plugin/head