Preface

Logstash is an open source log unified processing data collector, which belongs to the L in ELK and is widely used in the field of log collection.

The default log driver of docker is json-file , each container will generate a /var/lib/docker/containers/containerID/containerID-json.log locally, and the log driver supports extensions. This chapter mainly explains the use of logstash Collect docker logs.

Docker does not have the logstash driver, but the gelf-driven logs can be collected through the gelf input plugin of logstash.

premise

docker
Understanding logstash configuration
docker-compose

Prepare configuration file

docker-compose.yml

version: '3.7'

x-logging:
  &default-logging
  driver: gelf
  options:
    gelf-address: "udp://localhost:12201"
    mode: non-blocking
    max-buffer-size: 4m
    tag: "kafeidou.{
   
   {.Name}}"  #配置容器的tag,以kafeidou.为前缀,容器名称为后缀,docker-compose会给容器添加副本后缀,>如 logstash_1

services:

  logstash:
    ports:
      - 12201:12201/udp
    image: docker.elastic.co/logstash/logstash:7.5.1
    volumes:
      - ./logstash.yml:/usr/share/logstash/config/logstash.yml
      - /var/log/logstash:/var/log/logstash
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf

  logstash-worker:
    image: docker.elastic.co/logstash/logstash:7.5.1
    depends_on:
      - logstash
    logging:
      driver: "gelf"
      options:
        gelf-address: "udp://localhost:12201"

logstash.yml

http.host: "0.0.0.0"

logstash.conf

input {
 gelf{
  use_udp => true
  port_tcp => 12202
 }
}

 output {
   file {
        path => "/var/log/logstash/%{+yyyy-MM-dd-HH}/%{container_name}.log"
   }
 }

Because logstash need to have write permissions in the directory configuration, so you need to be ready to store log directory and give permission.
Create a directory

mkdir /var/log/logstash

Give permission, here is used for experimental demonstration, directly authorize 777

chmod -R 777 /var/log/logstash

Execute the command in the directory of the docker-compose.yml, logstash.conf and logstash.yml files:
docker-compose up -d

[root@master logstash]# docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Starting logstash_logstash_1 ... done
Starting logstash_logstash-worker_1 ... done

Logstash starts slowly, and the effect of my experiment is about 90 seconds, so it is more recommended to use fluentd to collect logs

Check the log directory, there should be a corresponding container log file:

[root@master logstash]# ls /var/log/logstash/
2020-02-16
[root@master logstash]# ls /var/log/logstash/2020-02-16/
logstash_logstash-worker_1.log

You can also download my file directly:

to sum up

For technical selection, fluentd is more recommended, why?

Fluentd is more lightweight and flexible, and currently belongs to CNCF, its activity and reliability have been improved.

Why also introduce logstash to collect docker logs?

If you are already using the ELK technology stack in a company or business, there may be no need to introduce a fluentd, but continue to use logstash to get through the docker log. Here is mainly a sharing, so that students who encounter this situation can have One more choice.

Use logstash as the docker log driver to collect logs