Grok 解析elk日志

<div id="article_content" class="article_content clearfix csdn-tracking-statistics" data-pid="blog" data-mod="popu_307" data-dsm="post">
 <div class="article-copyright">
 版权声明：本文为博主原创文章,未经博主允许禁止转载(http://blog.csdn.net/napoay) https://blog.csdn.net/napoay/article/details/62885899 </div>
 <div class="markdown_views">
 
 <svg xmlns="http://www.w3.org/2000/svg" style="display: none;"><path stroke-linecap="round" d="M5,0 0,2.5 5,5z" id="raphael-marker-block" style="-webkit-tap-highlight-color: rgba(0, 0, 0, 0);"></path></svg>
 <h1 id="一简介"><a name="t0"></a>一、简介</h1>

<blockquote>
Grok是迄今为止使蹩脚的、无结构的日志结构化和可查询的最好方式。Grok在解析 syslog logs、apache and other webserver logs、mysql logs等任意格式的文件上表现完美。
</blockquote>

Grok内置了120多种的正则表达式库，地址:<a href="https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns" rel="nofollow" target="_blank">https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns</a>。

下面是一条tomcat日志：

<pre class="prettyprint" name="code"><code class="hljs 1c has-numbering">83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li></ul></pre>

从filebeat中输出到logstash，配置如下:

<pre class="prettyprint" name="code"><code class="hljs php has-numbering">input {
beats {
port => "5043"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout { codec => rubydebug }
}</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li></ul></pre>

fileter中的<code>message</code>代表一条一条的日志，<code>%{COMBINEDAPACHELOG}</code>代表解析日志的正则表达式，COMBINEDAPACHELOG的具体内容见:<a href="https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd" rel="nofollow" target="_blank">https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd</a>。解析后:

<pre class="prettyprint" name="code"><code class="hljs php has-numbering">{
"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"offset" => 325,
"auth" => "-",
"ident" => "-",
"input_type" => "log",
"verb" => "GET",
"source" => "/path/to/file/logstash-tutorial.log",
"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
"@timestamp" => 2016-10-11T21:04:36.167Z,
"response" => "200",
"bytes" => "203023",
"clientip" => "83.149.9.216",
"@version" => "1",
"beat" => {
"hostname" => "My-MacBook-Pro.local",
"name" => "My-MacBook-Pro.local"
},
"host" => "My-MacBook-Pro.local",
"httpversion" => "1.1",
"timestamp" => "04/Jan/2015:05:13:42 +0000"
}</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li><li style="color: rgb(153, 153, 153);">8</li><li style="color: rgb(153, 153, 153);">9</li><li style="color: rgb(153, 153, 153);">10</li><li style="color: rgb(153, 153, 153);">11</li><li style="color: rgb(153, 153, 153);">12</li><li style="color: rgb(153, 153, 153);">13</li><li style="color: rgb(153, 153, 153);">14</li><li style="color: rgb(153, 153, 153);">15</li><li style="color: rgb(153, 153, 153);">16</li><li style="color: rgb(153, 153, 153);">17</li><li style="color: rgb(153, 153, 153);">18</li><li style="color: rgb(153, 153, 153);">19</li><li style="color: rgb(153, 153, 153);">20</li><li style="color: rgb(153, 153, 153);">21</li><li style="color: rgb(153, 153, 153);">22</li><li style="color: rgb(153, 153, 153);">23</li><li style="color: rgb(153, 153, 153);">24</li><li style="color: rgb(153, 153, 153);">25</li><li style="color: rgb(153, 153, 153);">26</li><li style="color: rgb(153, 153, 153);">27</li><li style="color: rgb(153, 153, 153);">28</li></ul></pre>

再比如,下面这条日志：

<pre class="prettyprint" name="code"><code class="hljs vbnet has-numbering">55.3.244.1 GET /index.html 15824 0.043</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li></ul></pre>

这条日志可切分为5个部分，<code>IP(55.3.244.1)</code>、<code>方法(GET)</code>、<code>请求文件路径(/index.html)</code>、<code>字节数(15824)</code>、<code>访问时长(0.043)</code>,对这条日志的解析模式(正则表达式匹配)如下:

<pre class="prettyprint" name="code"><code class="hljs css has-numbering">%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li></ul></pre>

写到filter中:

<pre class="prettyprint" name="code"><code class="hljs php has-numbering">filter {
grok {
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"}
}
}</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li></ul></pre>

解析后:

<pre class="prettyprint" name="code"><code class="hljs oxygene has-numbering">client: 55.3.244.1
method: GET
request: /index.html
bytes: 15824
duration: 0.043</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li></ul></pre>

<h1 id="三解析任意格式日志"><a name="t2"></a>三、解析任意格式日志</h1>

解析任意格式日志的步骤：

<ol>
<li>先确定日志的切分原则，也就是一条日志切分成几个部分。</li>
<li>对每一块进行分析，如果Grok中正则满足需求，直接拿来用。如果Grok中没用现成的，采用自定义模式。</li>
<li>学会在<a href="http://grokdebug.herokuapp.com/" rel="nofollow" target="_blank">Grok Debugger</a>中调试。</li>
</ol>

下面给出例子，来两条日志:

<pre class="prettyprint" name="code"><code class="hljs css has-numbering">2017-03-07 00:03:44,373 4191949560 [ CASFilter.java:330:DEBUG] entering doFilter()

2017-03-16 00:00:01,641 133383049 [ UploadFileModel.java:234:INFO ] 上报内容准备写入文件</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li></ul></pre>

切分原则:

<pre class="prettyprint" name="code"><code class="hljs css has-numbering">2017-03-16 00:00:01,641:时间
133383049：编号
UploadFileModel.java:java类名
234:代码行号
INFO：日志级别
entering doFilter()：日志内容
</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li><li style="color: rgb(153, 153, 153);">2</li><li style="color: rgb(153, 153, 153);">3</li><li style="color: rgb(153, 153, 153);">4</li><li style="color: rgb(153, 153, 153);">5</li><li style="color: rgb(153, 153, 153);">6</li><li style="color: rgb(153, 153, 153);">7</li></ul></pre>

前五个字段用Grok中已有的，分别是<code>TIMESTAMP_ISO8601</code>、<code>NUMBER</code>、<code>JAVAFILE</code>、<code>NUMBER</code>、<code>LOGLEVEL</code>，最后一个采用自定义正则的形式，日志级别的]之后的内容不论是中英文，都作为日志信息处理，使用自定义正则表达式子的规则如下:

<pre class="prettyprint" name="code"><code class="hljs clojure has-numbering">(?<field_name>the pattern here)</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li></ul></pre>

最后一个字段的内容用info表示，正则如下:

上面两条日志对应的完整的正则如下，其中<code>\s*</code>用于剔除空格。

<pre class="prettyprint" name="code"><code class="hljs ruby has-numbering">\s*%{TIMESTAMP_ISO8601:time}\s*%{NUMBER:num} \[\s*%{JAVAFILE:class}\s*\:\s*%{NUMBER:lineNumber}\s*\:%{LOGLEVEL:level}\s*\]\s*(?<info>([\s\S]*))</code><ul class="pre-numbering" style=""><li style="color: rgb(153, 153, 153);">1</li></ul></pre>

正则解析容易出错，强烈建议使用Grok Debugger调试，姿势如下。

<ol>
<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html" rel="nofollow" target="_blank">plugins-filters-grok</a></li>
<li><a href="https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html" rel="nofollow" target="_blank">Parsing Logs with Logstash</a></li>
</ol> </div>
<link href="https://csdnimg.cn/release/phoenix/mdeditor/markdown_views-8cccb36679.css" rel="stylesheet">
</div>

猜你喜欢