elasticsearch logstash filter使用测试

---

elasticsearch logstash filter使用测试

**********************************

grok 过滤器

语法：%{pattern_name：field_name}

pattern_name：正则表达式，匹配message中的文本

field_name：匹配文本的name，value为匹配的文本内容

****************************

常用内置表达式：官网链接

NUMBER、(?:%{BASE10NUM})：10进制整数字符串

BASE16NUM ：16进制整数字符串

WORD、\b\w+\b：匹配字符串

SPACE、\s*：空格

DATA、.*?：任意多个字符

GREEDYDATA、.*：任意多个字符

IP：匹配ip4或者ip6

HOSTNAME：匹配主机名，如www.baidu.com

TIMESTAMP_ISO8601：匹配时间戳

LOGLEVEL ：日志输出级别，如info、warn等

****************************

常用设置

pattern_definitions：设置自定义正则表达式，hash形式

    filter {
      grok {
        pattern_definitions => { 
          "pattern_name" => "pattern_value",
          "pattern_name2" => "pattern_value2"
        }
      }
    }

match：解析message，提取内容存储，hash形式

    filter {
      grok {
        match => { 
          "message" => "%{pattern_name:field_name}"
        }
      }
    }

id：设置过滤器的id，string形式

    filter {
      grok {
        id => "ABC"
      }
    }

add_field：添加字段，hash形式

    filter {
      grok {
        pattern_definitions => { 
          add_field => {
          "foo_%{somefield}" => "Hello world, from %{host}"
          "new_field" => "new_static_value"
          }
        }
      }
    }

remove_field：删除字段，array形式

    filter {
      grok {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

add_tag：添加tag，array形式

filter {
      grok {
        add_tag => [ "foo_%{somefield}", "tag_1"]
      }
    }

remove_tag：删除tag

    filter {
      grok {
        remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
      }
    }

****************************

示例

文本：

hello world www.baidu.com

管道配置

input {
  file {
    path => "/usr/share/logstash/logs/hello2.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    id => "grok"

    pattern_definitions => {
      "p1" => "\S*\s*\S*"
    }

    match => {
      "message" => "%{p1:word}\s*%{HOSTNAME:ip}"
    }

    add_field => {
      "field" => "%{word}"
      "field2" => "hello2"
    }

    add_tag => [
      "tag","tag2"
    ]

    remove_field => ["path"]
  }

  geoip {
    source => "ip"
  }
}

output {
  stdout { codec => rubydebug }
}

控制台输出

             

**********************************

geoip 过滤器：解析ip地址、hostname相关信息

文本

www.baidu.com
www.jd.com

logstash管道配置

input {
  file {
    path => "/usr/share/logstash/logs/hello3.log"
    start_position => "beginning"
  }
}

filter {
  grok {
    match => {
      "message" => "%{HOSTNAME:ip}"
    }
  }

  geoip {
    source => "ip"
  }
}

output {
  stdout { codec => rubydebug }
}

控制台输出