Loki+Grafana (external) collects Kubernetes (K8s) cluster (based on containerd)

1. Introduction to Loki

---

1 Introduction

Loki is an open source, distributed log aggregation system launched by Grafana Labs. Loki is designed to efficiently process large-scale log data and have good scalability. The biggest advantage of Loki is its low resource footprint and efficient query speed. This is because Loki does not need to index when processing log data, but stores the data in an in-memory data structure similar to a hash table, which can quickly locate and retrieve log data. In addition, Loki can be integrated with Prometheus to achieve more powerful monitoring functions.

2. Components

• Distributor: is a stateless component responsible for obtaining log data and forwarding it to ingester. Distributor preprocesses the data, checks its validity, and ensures it comes from a configured tenant, which helps the system scale and protects it from potential DoS attacks. Here Grafana does a good job of explaining how Promtail (the recommended distribution agent) handles data.
• ingester: It is a key component of Loki architecture. Data received from distributors is written to the cloud native storage service by Ingredients. Ingesters also collaborate with queries to return in-memory data in response to read requests.
• Queriers: Responsible for parsing LogQL query requests and obtaining data from ingesters or persistent storage.
• query fronted: Optional component - provides API interface, which can be used to accelerate read processing. This component optimizes read processing by queuing read requests, splitting large requests into multiple smaller requests, and caching data.

2. Experimental environment

---

1. k8s environment

Version v1.26.5, the container is containerd
binary installation Kubernetes (K8s) cluster (based on containerd) - installation tutorial from scratch (with certificate)

CPU name	IP	system version	install service
master01	10.10.10.21	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master02	10.10.10.22	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
master03	10.10.10.23	rhel7.5	nginx、etcd、api-server、scheduler、controller-manager、kubelet、proxy
node01	10.10.10.24	rhel7.5	nginx、kubelet、proxy
node02	10.10.10.25	rhel7.5	nginx、kubelet、proxy

2. Prometheus+Grafana environment

Prometheus+Grafana monitoring system

CPU name	IP	system version	install service
jenkins	10.10.10.10	rhel7.5	docker、Prometheus、Grafana

3. Loki installation

---

https://github.com/grafana/loki/releases
https://grafana.com/docs/loki/latest/installation/docker/

1. Download the image

[root@jenkins ~]# docker pull grafana/loki:2.8.2
[root@jenkins ~]# docker pull grafana/promtail:2.8.2

Push to local harbor:

[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker tag grafana/loki:2.8.2 harbor.wielun.com/library/grafana/promtail:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/loki:2.8.2
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.8.2

2. Start loki

[root@jenkins ~]# mkdir -p /etc/loki/conf
[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/cmd/loki/loki-local-config.yaml -O loki-config.yaml
[root@jenkins ~]# mv loki-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name loki \
--privileged=true \
--restart always -p 3100:3100 \
-v /etc/loki/conf:/mnt/config \
grafana/loki:2.8.2 \
-config.file=/mnt/config/loki-config.yaml

[root@jenkins ~]# cat /etc/loki/conf/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

3. Start promtail

[root@jenkins ~]# wget https://raw.githubusercontent.com/grafana/loki/v2.8.2/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
[root@jenkins ~]# mv promtail-config.yaml /etc/loki/conf/
[root@jenkins ~]# docker run -d --name promtail \
  --privileged=true \
  --restart always \
  -v /etc/loki/conf/:/mnt/config \
  -v /var/log:/var/log \
  grafana/promtail:2.8.2 \
  -config.file=/mnt/config/promtail-config.yaml

[root@jenkins ~]# cat /etc/loki/conf/promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://10.10.10.10:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

4. Test results

http://10.10.10.10:3100/metrics
http://10.10.10.10:3100/ready

5. Install promtail on winserver

(1) Download promtail

(2) Configure promtail-config.yaml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: D:/loki/config/positions.yaml

clients:
  - url: http://10.10.10.10:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: D:/loki/logs/*.out

(3) Start

./promtail-windows-amd64.exe --config.file=config/promtail-config.yaml

4. Grafana view

---

grafana can use 13639 template

1. Add Loki data source

2. Check the local log

5. Collect K8s logs

---

Official website address: https://grafana.com/docs/loki/latest/clients/promtail/installation/

1. Upload the image

It was found that the image startup using 2.8.2 was unsuccessful, so version 2.5.0 is used here.

[root@jenkins ~]# docker pull grafana/promtail:2.5.0
[root@jenkins ~]# docker tag grafana/promtail:2.5.0 harbor.wielun.com/library/grafana/promtail:2.5.0
[root@jenkins ~]# docker push harbor.wielun.com/library/grafana/promtail:2.5.0

2. Create Endpoints

[root@master01 ~]# kubectl create ns logging
[root@master01 ~]# cat loki.yaml
apiVersion: v1
kind: Endpoints
metadata:
  name: loki
  namespace: logging
subsets:
  - addresses:
    - ip: 10.10.10.10
    ports:
    - port: 3100

---
apiVersion: v1
kind: Service
metadata:
  name: loki
  namespace: logging
spec:
  clusterIP: None
  ports:
  - name: loki
    port: 3100
    protocol: TCP
    targetPort: 3100
  type: ClusterIP

[root@master01 ~]# kubectl apply -f loki.yaml

3. Create promtail

[root@master01 ~]# cat promtail.yaml
--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
  namespace: logging
  labels:
    app: promtail
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: harbor.wielun.com/library/grafana/promtail:2.5.0
        imagePullPolicy: IfNotPresent
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env:
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
  namespace: logging
  labels:
    app: promtail
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
      - url: http://loki.logging.svc.cluster.local:3100/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
  labels:
    app: promtail
  namespace: logging
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount
  labels:
    app: promtail
  namespace: logging

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
  labels:
    app: promtail
  namespace: logging
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: default
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

[root@master01 ~]# kubectl apply -f promtail.yaml

4. View the results

[root@master01 ~]# kubectl get pod -n logging
NAME                       READY   STATUS    RESTARTS   AGE
promtail-daemonset-2r8sv   1/1     Running   0          56s
promtail-daemonset-f4p8q   1/1     Running   0          56s
promtail-daemonset-tddsf   1/1     Running   0          56s
promtail-daemonset-wdn47   1/1     Running   0          56s
promtail-daemonset-gry5e   1/1     Running   0          56s

5. Use of dashboard

https://grafana.com/grafana/dashboards/?dataSource=loki