K8S+DevOps architect practical course | k8s log collection architecture

Video source: Station B "Docker&k8s Tutorial Ceiling, Absolutely the best one taught by Station B, this set of learning k8s to get all the core knowledge of Docker is here"

Organize the teacher's course content and test notes while studying, and share them with everyone. Any infringement will be deleted. Thank you for your support!

Attach a summary post: K8S+DevOps Architect Practical Course | Summary

---

Generally divided into three ways:

• Use a node-level logging agent running on each node.
• In the pod of the application, there are sidecar containers dedicated to logging.
• Push logs directly from the application to the logging backend.

Use node-level logging agent

Container log driver:

https://docs.docker.com/config/containers/logging/configure/

View the driver of the current docker host:

$ docker info --format '{
   
   {.LoggingDriver}}'

json-file format, docker will save the standard and error output as a file of the host by default, and the path is:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

And you can set log rotation:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "labels": "production_status",
    "env": "os,customer"
  }
}

Advantage:

• Easy to deploy, just use the DaemonSet type controller to deploy the agent
• Minimal impact on business applications, non-intrusive

Disadvantages:

• Only standard and error output can be collected, and the file logs in the container cannot be collected temporarily

Use sidecar container and logging proxy

• Method 1: The sidecar container sends application logs to its own standard output.

Idea: Start a sidecar container in the pod, spit out the log files in the container to the standard output, and collect them by the log collection agent in the host.

$ cat count-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-1
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-2
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}
    
$ kubectl apply -f counter-pod.yaml
$ kubectl logs -f counter -c count-log-1

• Advantages: It can realize the internal log collection of the container, which is less intrusive to business applications
• Disadvantages: Each business pod needs to be modified once, adding a log write, which has a certain impact on disk usage
• Method 2: The sidecar container runs a log proxy, and configures the log proxy to collect logs from the application container.

Idea: Start a log collection component (such as fluentd) directly in the business pod by using sidecar, so that the log collection can collect the logs in the container as local files.

Advantage: There is no need to store logs to the host, and local logs can be collected completely

Disadvantage: Each business application starts an additional log agent, which brings additional resource consumption

Expose the log directory directly from the application

Enterprise log solution selection

At present, the most recommended is to use a node-level log proxy.

Solution 1: Self-developed solution, implement a self-developed log collection agent, general ideas:

• For the standard output and error output of the container, use the conventional method to monitor the container output path in the host
• For the log files inside the container, configure unified environment variables in the container, such as LOG_COLLECT_FILES, and specify the log directory and files to be collected in the container. When the agent starts, mount the docker.sock file and the root path of the disk to monitor docker container creation, Delete the event, find out the container storage, environment variables, k8s attributes and other information through the docker API. The container configured with the LOG_COLLECT_FILES environment variable, find the corresponding file path in the host according to the log path in env, and then generate the collected configuration file agent In cooperation with open source log collection tools (Fluentd or filebeat, etc.), the agent is responsible for delivering configuration to the collection tool and reloading the process

fluentd-pilot

Solution 2: Logs are collected using an open source Agent (EFK solution), which has a wide range of applications and can meet the needs of most log collection and display.