在 Filebeat +Kafka + Logstash + ElasticSearch +Kibana +解析日志文件实例(一)上一节中我们跑通了整个架构的数据链路，后面将持续更新，逐步细化其中的几个问题，比如：
1、Filebeat如何在采集数据时只提取有用的数据发给kafka？
2、logstash消费了kafka消息后，如何格式化数据发给elasticsearch？
3、kibana又如何用直观的显示我们希望看到的日志报表？
今天这篇文章将解决第一个问题，Filebeat如何在采集数据时只提取有用的数据发给kafka的。
核心操作就是配置filebeat.yml文件：
官方配置说明：https://www.elastic.co/guide/en/beats/filebeat/index.html（选择版本）
Filebeat7.5版本配置页面地址：https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html
也可以参考我的《filebeat.yml配置文件详细说明》
一、过滤我们需要的信息，日志文件内容假设如下：
2017/06/22 11:26:30 [ERROR] 26067#0: *17918 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.32.17, server: localhost, request: "GET /wss/ HTTP/1.1", upstream: "http://192.168.12.106:8010/", host: "192.168.12.106"
2017/06/22 11:26:30 [info] 26067#0:
2017/06/22 12:05:10 [error] 26067#0: *17922 open() "/data/programs/nginx/html/ws" failed (2: No such file or directory), client: 192.168.32.17, server: localhost, request: "GET /ws HTTP/1.1", host: "192.168.12.106"
1、使用include_lines参数过滤error行信息：不区分大小写
扫描二维码关注公众号，回复：
12674264 查看本文章
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
include_lines: ['[ERROR]']2、查看kibana信息，可以看到error的两行被采集，且不区'error'分大小写
补充：今天测试的时候又发现filebeat里面区分大小写，而且还不能识别特殊字符，比如圆括号，中括号等，要改成下面的才行：
include_lines: ['\[ERROR\]','\[error\]']
3、对于java应用常见的错误是多行的，我们来一个多行的错误采集测试，使用multiline与include_lines结合使用,日志如下
[2019-11-20 11:26:30.573][ERROR] 26067#0: *17918 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.32.17, server: localhost, request: "GET /wss/ HTTP/1.1", upstream: "http://192.168.12.106:8010/", host: "192.168.12.106"
[2019-11-20 11:26:30.573][info] 26067#0:
[2019-11-20 12:05:10.123][error] 26067#0: *17922 open() "/data/programs/nginx/html/ws" failed (2: No such file or directory), client: 192.168.32.17, server: localhost, request: "GET /ws HTTP/1.1", host: "192.168.12.106"
[2019-11-20 20:25:22.573][DubboServerHandler-192.168.0.5:20904-thread-3][ERROR][?][]:{conn-10005, pstmt-20003} execute error. select count(1)
from (
select t.RETAIL_INVOICE_ID as retailInvoiceId, t.RETAIL_INVOICE_CODE as retailInvoiceCode, t.BILL_CODE as billCode, t.CUSTOMER_ID as customerId, t.CURRENCY as currency
, t.SALE_CHANNEL as saleChannel, t.REMARK as remark, t.COMPANY_ID as companyId, t.CREATED_BY as createdBy, t.CREATED_DATE as createdDate
, t.MODIFIED_BY as modifiedBy, t.MODIFIED_DATE as modifiedDate, t.RD_STATUS as rdStatus, t.DATA_ID as dataId, t.COMPANY_NAME as companyName
, case T.DATA_FROM
when 1 then 'TMS'
when 2 then '接口'
when 3 then '手动导入'
else null
end as dataFromName
where T.RD_STATUS = 1
and TSP.BL_SCM = 1
and t.DATE_INVOICE >= to_date('2019-11-13 00:00:00', 'yyyy-mm-dd hh24:mi:ss')
and t.DATE_INVOICE <= to_date('2019-11-20 23:59:59', 'yyyy-mm-dd hh24:mi:ss')
order by T.CREATED_DATE desc
) TOTAL
java.sql.SQLSyntaxErrorException: ORA-00920: invalid relational operator
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:206) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:455) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:413) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:1034) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1186) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3387) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3431) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) ~[mybatis-generator-core-1.3.2-ztd.jar:?]
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2830) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(FilterEventAdapter.java:465) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2827) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.filter.FilterEventAdapter.preparedStatement_executeQuery(FilterEventAdapter.java:465) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.filter.FilterChainImpl.preparedStatement_executeQuery(FilterChainImpl.java:2827) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.proxy.jdbc.PreparedStatementProxyImpl.executeQuery(PreparedStatementProxyImpl.java:181) ~[druid-1.1.5.jar:1.1.5]
at com.alibaba.druid.pool.DruidPooledPreparedStatement.executeQuery(DruidPooledPreparedStatement.java:228) ~[druid-1.1.5.jar:1.1.5]
at com.baomidou.mybatisplus.plugins.PaginationInterceptor.count(PaginationInterceptor.java:189) ~[mybatis-plus-2.0.2.jar:?]
at com.baomidou.mybatisplus.plugins.PaginationInterceptor.intercept(PaginationInterceptor.java:156) ~[mybatis-plus-2.0.2.jar:?]
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61) ~[mybatis-3.4.2.jar:3.4.2]
at com.sun.proxy.$Proxy184.query(Unknown Source) ~[?:?]
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148) ~[mybatis-3.4.2.jar:3.4.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_171]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_171]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_171]
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:433) ~[mybatis-spring-1.3.1.jar:1.3.1]
at com.sun.proxy.$Proxy57.selectList(Unknown Source) ~[?:?]
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:238) ~[mybatis-spring-1.3.1.jar:1.3.1]
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:135) ~[mybatis-3.4.2.jar:3.4.2]
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:75) ~[mybatis-3.4.2.jar:3.4.2]
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59) ~[mybatis-3.4.2.jar:3.4.2]
at com.sun.proxy.$Proxy131.findPage(Unknown Source) ~[?:?]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl.findPage(TabRetailInvoiceServiceImpl.java:83) ~[classes/:?]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(<generated>) ~[classes/:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:62) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:47) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$7183f17b.findPage(<generated>) ~[classes/:?]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(<generated>) ~[classes/:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$62d14ac0.findPage(<generated>) ~[classes/:?]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$FastClassBySpringCGLIB$$2020b273.invoke(<generated>) ~[classes/:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) ~[spring-core-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673) ~[spring-aop-4.3.16.RELEASE.jar:4.3.16.RELEASE]
at com.zhitengda.dolphin.modular.service.impl.invoice.retailinvoice.TabRetailInvoiceServiceImpl$$EnhancerBySpringCGLIB$$62d14ac0.findPage(<generated>) ~[classes/:?]
at com.alibaba.dubbo.common.bytecode.Wrapper21.invokeMethod(Wrapper21.java) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:64) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:61) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:132) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:69) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:98) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:98) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52) ~[dubbo-2.5.7.jar:2.5.7]
at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:81) ~[dubbo-2.5.7.jar:2.5.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
4、增加multiline配置如下
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#不以[开头的行都合并到上一行的末尾
multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#默认是false，匹配pattern的行合并到上一行；true，不匹配pattern的行合并到上一行
multiline.negate: true
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#after 或 before，合并到上一行的末尾或开头
multiline.match: after5、清除registry目录后重启filebeat，检查kibana数据
我们主要查看最后一个error命中内容，内容较长，展开如下：
数据采集正确。
下面贴上我的filebeat.yml文件
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
#开启监视，不开不采集
enabled: true
# 指定采集的路径，基于全局的路径（可用通配符，但不递归）.
paths:
#- /var/log/*.log (linux例子)
#- c:\programdata\elasticsearch\logs\* (windows 例子)
- D:\localinstall\ELK\data-log\*.log
# 排除行, 要匹配的正则表达式的列表. 它将从列表中删除与任何正则表达式匹配的行
#exclude_lines: ['^DBG']
# 包含行. 要匹配的正则表达式的列表. 它从列表中导出与任何正则表达式匹配的行
include_lines: ['[ERROR]']
# 排除文件. 要匹配的正则表达式的列表。 Filebeat 将从列表中删除与任何正则表达式匹配的文件。默认情况下, 不会删除任何文
#exclude_files: ['.gz$']
# 可选的附加字段。可以随意选取这些字段, 以便将附加信息添加到已爬网日志文件中以进行筛选
#fields:
# level: debug
# review: 1
### 多行选项
# 可用于跨越多行的日志消息。这对于 Java 堆栈跟踪或 C 行继续很常见
# 必须匹配的 regexp 模式。示例模式匹配所有开始 [
multiline.pattern: ^\[
# 定义在模式下设置的模式是否应该被否定。默认值为 false.
multiline.negate: true
# 匹配可以设置为 "后" 或 "之前"。它用于定义是否应将行追加到模式 (不匹配) 之前或之后, 或者只要模式不匹配 (基于否定.
# Note: 后是等同于前和前是等价于下 Logstash
multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# 设置为true来启用配置重载
reload.enabled: false
# 检查路径下的文件更改的期间（多久检查一次）
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
#tags: ["service-X", "web-tier"]
#fields:
# env: staging
#============================== Dashboards =====================================
# 这些设置控制将示例仪表板加载到 Kibana 索引。默认情况下, 加载仪表板被禁用, 可以通过在此处设置选项, 或者使用 "安装" CLI 标志或 "安装" 命令来启用。
setup.dashboards.enabled: false
# 从何处下载仪表板存档的 URL。默认情况下, 此 URL 具有一个基于节拍名称和版本计算的值。对于发布的版本, 此 URL 指向 artifacts.elastic.co 网站上的仪表板存档.
#setup.dashboards.url:
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Kafka output ------------------------------
output.kafka:
# Array of hosts to connect to.
hosts: ["127.0.0.1:9092","127.0.0.1:9093","127.0.0.1:9094"]
topic: filebeat_log
enabled: true
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~