Docker container log management Fluentd

With the widespread use of Docker containers, it is very painful for large-scale container clusters to not be able to centrally manage logs. If the application log cannot be collected, managed and retrieved reasonably, when the application fails, it will not be able to trace the cause of the problem well. This article mainly introduces Docker to configure Fluentd for log management

1. Quickly start Fluentd

# docker run -d -p 24224:24224 -p 24224:24224/udp -v /data:/fluentd/log fluent/fluentd

At this point, data.<fluentd container id>.log will be generated in the host/data directory, and all collected log files will be stored here.

2. Start a test container to see if the logs are stored in fluentd

# docker run -d --log-driver fluentd --log-opt fluentd-address=localhost:24224 --log-opt tag="nginx-test" --log-opt fluentd-async-connect  --name nginx-test -p 8080:80 nginx

--log- driver: configure log driver
 --log- opt: configure log-related parameters

fluentd - address: fluentd service address
fluentd -async-connect: fluentd-docker is set asynchronously to avoid the Docker container hanging up after fluentd hangs

After configuration, visit the nginx page, and the following log will appear every time you refresh

2018-05-03T07:21:55+00:00    nginx-test    {
    "container_name": "/nginx-test",
    "source": "stdout",
    "log": "172.96.247.193 - - [03/May/2018:07:21:55 +0000] \"GET / HTTP/1.1\" 304 0 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36\" \"-\"",
    "container_id": "0cd8d7d68fe233238679b31327d53ad0fffe5b419f1847ad9140db063dded7f3"
}

3. Docker global log configuration

# cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://zcg96r7h.mirror.aliyuncs.com"],
  "log-driver": "fluentd",
   "log-opts": {
     "fluentd-address": "127.0.0.1:24224"
   }
}

#Notice:

a. After using fluentd, you cannot use docker logs to view;

b. The container log created before configuring fluentd will not be written to Fluentd. If you want to store it, you need to rebuild the container;

c. After configuring fluentd globally, if the fluentd service is abnormal, the container cannot be started;

# docker run --rm nginx
docker: Error response from daemon: failed to initialize logging driver: dial tcp 127.0.0.1:24224: getsockopt: connection refused.

Start another test container, and you can see that the logs are also stored in Fluentd

Reference documentation:

1、https://www.fluentd.org/guides/recipes/docker-logging

2. Installation documentation: https://docs.fluentd.org/v1.0/articles/quickstart#step-1:-installing-fluentd

3、Docker配置Fluentd：https://docs.docker.com/config/containers/logging/fluentd/

Docker container log management Fluentd

Guess you like