elasticsearch filebeat采集日志输出到elasticsearch

---

elasticsearch filebeat采集日志输出到elasticsearch

使用filebeat采集日志文件，将日志输出到logstash，logstash输出到elasticsearch

*****************************

日志文件：my.log

2020-01-12 10:06:14.549  INFO 9356 --- [main] com.example.demo.DemoApplication         : Starting DemoApplication on LAPTOP-D73GD8TE with PID 9356 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
2020-01-12 10:06:14.555  INFO 9356 --- [main] com.example.demo.DemoApplication         : No active profile set, falling back to default profiles: default
2020-01-12 10:06:16.775  INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-01-12 10:06:16.787  INFO 9356 --- [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-01-12 10:06:16.787  INFO 9356 --- [main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.29]
2020-01-12 10:06:16.967  INFO 9356 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-01-12 10:06:16.967  INFO 9356 --- [main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 2314 ms
2020-01-12 10:06:17.422  INFO 9356 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-01-12 10:06:17.710  INFO 9356 --- [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-01-12 10:06:17.781  INFO 9356 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-01-12 10:06:17.787  INFO 9356 --- [main] com.example.demo.DemoApplication         : Started DemoApplication in 4.119 seconds (JVM running for 6.441)
2020-01-12 10:06:19.332  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-01-12 10:06:19.333  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-01-12 10:06:19.342  INFO 9356 --- [RMI TCP Connection(5)-192.168.57.1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 8 ms
2020-01-12 10:09:21.353  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : Starting DemoApplicationTests on LAPTOP-D73GD8TE with PID 3104 (started by 28401 in E:\java\IdeaProjects\springboot hello-world)
2020-01-12 10:09:21.355  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : No active profile set, falling back to default profiles: default
2020-01-12 10:09:23.758  INFO 3104 --- [main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2020-01-12 10:09:24.435  INFO 3104 --- [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2020-01-12 10:09:24.516  INFO 3104 --- [main] com.example.demo.DemoApplicationTests    : Started DemoApplicationTests in 3.886 seconds (JVM running for 5.747)
2020-01-12 10:09:25.677  INFO 3104 --- [SpringContextShutdownHook] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

*****************************

相关配置

filebeat配置

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

output.logstash:
  hosts: ["172.18.0.32:5044"]

logstash管道配置

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch { 
    hosts => ["172.18.0.33:9200"]
    index => "log-%{+yyyy.MM.dd}"
  }
}

 

*****************************

docker 创建容器

docker run -it --net fixed --ip 172.18.0.31 \
-v /usr/elasticsearch/filebeat/config/filebeat3.yml:/usr/share/filebeat/filebeat.yml \
-v /usr/elasticsearch/filebeat/logs:/usr/share/filebeat/logs \
--name filebeat docker.elastic.co/beats/filebeat:7.5.1

docker run -it --net fixed --ip 172.18.0.32 -p 5044:5044 \
-v /usr/elasticsearch/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v /usr/elasticsearch/logstash/config/logstash3.conf:/usr/share/logstash/pipeline/logstash.conf \
--name logstash docker.elastic.co/logstash/logstash:7.5.1

docker run -it --net fixed --ip 172.18.0.33 -p 9201:9200 -p 9301:9300  \
-e ES_JAVA_OPTS="-Xms512m -Xmx512m"  \
-v /usr/elasticsearch/single/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
--name es-single2 elasticsearch:7.5.1

docker run -it --net fixed --ip 172.18.0.34 -p 5601:5601 \
-e ELASTICSEARCH_HOSTS="http://172.18.0.33:9200" \
--name kibana docker.elastic.co/kibana/kibana:7.5.1

*************************

kibana显示