Kubernetes logging system upstart Loki-Stack

Kubernetes logging system upstart Loki-Stack

Scofield novice operation and maintenance talk

Introduction to Loki

---

Grafana Loki is a set of components that can form a fully functional logging stack.
Unlike other logging systems, Loki is built on the idea of ​​indexing only metadata about logs: tags (just like Prometheus tags). Then, the log data itself is compressed and stored in blocks in object storage (such as S3 or GCS), or even stored locally in the file system. Small indexes and highly compressed blocks simplify operations and greatly reduce Loki's cost.
Compared to EKL, Loki is very lightweight; after using Loki, ELK suddenly loses its fragrance! Haha~~~

Loki-stack components

---

Put two pictures first

• Promtail
  Promtail is a log collection tool used to send container logs to Loki or Grafana services. This tool mainly includes discovering collection targets and adding Label tags to log streams, and then sending them to Loki. In addition, Promtail's service discovery is based on Prometheus The service discovery mechanism is implemented.
• Loki
  Loki is a log aggregation system that can scale horizontally, is highly available, and supports multi-tenancy. It uses the same service discovery mechanism as Prometheus, adding tags to the log stream instead of building a full-text index. Therefore, the logs received from Promtail and the applied metrics have the same label set. It not only provides better context switching between logs and indicators, but also avoids full-text indexing of logs.
• Grafana
  Grafana is an open source platform for monitoring and visual observation. It supports very rich data sources. In the Loki technology stack, it is specifically used to display time series data from data sources such as Prometheus and Loki. It also allows operations such as query, visualization, alarm, etc., which can be used to create, explore and share data Dashboard.
  Next, let’s get started and experience Loki-stack.

Use helm to deploy loki-stack

---

1. Prerequisite, you need to have a k8s cluster and install the helm tool
2. Add the helm source

helm repo add loki https://grafana.github.io/loki/charts

下载到本地，也可以直接安装，因为需要修改存储类及其他一些配置，我习惯下载到本地，方便修改调试
helm fetch loki/loki-stack
tar -zxf loki-stack-0.38.1.tgz
cd loki-stack/
[root@ loki-stack]# ls -al
drwxr-xr-x.  7 root root   85 Jul  2 07:25 charts
-rwxr-xr-x.  1 root root  369 Jul  1 01:46 Chart.yaml
-rwxr-xr-x.  1 root root  342 Jul  1 01:46 .helmignore
-rwxr-xr-x.  1 root root 1662 Jul  1 01:46 README.md
-rwxr-xr-x.  1 root root  533 Jul  1 01:46 requirements.lock
-rwxr-xr-x.  1 root root  595 Jul  1 01:46 requirements.yaml
drwxr-xr-x.  3 root root   80 Jul  2 08:46 templates
-rwxr-xr-x.  1 root root  206 Jul 16 00:53 values.yaml

3. Modify the values.yaml file of loki-stack, I only installed loki and promtail here, and I use the prometheus-operator that the cluster has deployed for grafana and prometheus

[root@ ]# cat values.yaml
loki:
  enabled: true
promtail:
  enabled: true

4. Modify loki's values.yaml,
mainly modify storageClass and persistence size, and open serviceMonitor to monitor loki

[root@  loki-stack]# cd charts/loki/
[root@  loki]# ll
total 20
-rw-r--r-- 1 root root  367 Nov 28 00:31 Chart.yaml
-rw-r--r-- 1 root root 1846 Nov 28 00:31 README.md
drwxr-xr-x 2 root root 4096 Dec  1 09:14 templates
-rw-r--r-- 1 root root 6695 Dec  1 09:15 values.yaml

persistence:
  enabled: true
  storageClass: "rbd"
  accessModes:
  - ReadWriteOnce
  size: 100Gi
  annotations: {}

serviceMonitor:
  enabled: true
  interval: ""
  additionalLabels: {}
  annotations: {}
  # scrapeTimeout: 10s

5. Modify the values.yaml of promtail.
If you want to add more log directories, you can add them here

# Extra volumes to scrape logs from
volumes:
- name: docker
  hostPath:
    path: /var/lib/docker/containers
- name: pods
  hostPath:
    path: /var/log/pods

6. Perform deployment

cd loki-stack
kubectl create ns loki
helm install loki -n loki -f values.yaml .

[root@ loki-stack]# helm ls -n loki
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
loki    loki            1               2020-12-01 09:15:22.971437761 +0800 CST deployed        loki-stack-2.1.0        v2.0.0

7. Log in to the existing grafana interface, add loki source,
first get the service address of loki

[root@ loki-stack]# kubectl get svc -n loki
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
loki            NodePort    10.101.129.255   <none>        3100:30439/TCP   92m
loki-headless   ClusterIP   None             <none>        3100/TCP         92m

After filling in the information, click Save&Test to confirm that grafana can connect to loki
8. Click Explore, you can use LogQL to query the log, as shown below

Of course, you can also import templates made by others. More

dashboards can be viewed on the official website https://grafana.com/grafana/dashboards. PS: Articles will be synchronized to dev.kubeops.net