[Big Data] LPG log collection solution (official website entry case)

This article introduces LPG, LPG experiments, and the official Getting started case. It is very interesting and the process will take several hours. Come and experience it together!

The log collection solution I have been using before is ELK, which often takes up several gigabytes of memory. Some servers with poor configurations are a bit overwhelmed! Recently I discovered a lightweight log collection solution: Loki+Promtail+Grafana (LPG for short). A few hundred MB of memory is enough, and the interface is quite good. I recommend it to everyone!

Preface: The environment is a Linux machine, not a mac or windows!

1. Introduction to LPG

The LPG log collection solution occupies very little memory, is economical and efficient! It does not index logs like the ELK logging system, but instead sets a set of tags for each log stream. Its core components are introduced below:

• Promtail: Log collector, a bit like Filebeat, can collect logs in log files and push the collected data to Loki.

• Loki: aggregates and stores log data and can be used as a data source for Grafana to provide visual data for Grafana.

• Grafana: Obtain log information from Loki and display it visually.

Explanation of the above picture:

1. Each server needs to be deployed Promtail. The responsibility is to monitor and collect the logs of this server and push them to the Loki storage service. The default is the monitoring and collection /var/logdirectory .

2. Loki is responsible for log storage

3. Grafana is data visualization and is responsible for reading logs in Loki.

2. Installation

To implement this log collection solution, you need to install services such as Loki, Promtail, and Grafana. It docker-composeis very convenient to install them directly.

We adopt the Install with Docker Compose solution from the official website . The following is a brief introduction to the official website solution.

1. First you need to install Docker Compose (please install it yourself)

2. Download the docker-compose script and execute it

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up

Because the website is inaccessible by default abroad, the author provided the script docker-compose.yaml that I downloaded for direct use. Readers can also use the reference blog to try to solve the problem of inaccessibility of raw.githubusercontent.com . The complete script and instructions are also provided in the appendix at the end of this article.

3. After running successfully, you can use docker psthe command to view 3 services

[root@server123 ~]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                       NAMES
a519d567e6a4   grafana/promtail:2.8.0   "/usr/bin/promtail -…"   About an hour ago   Up About a minute                                               plg-promtail-1
c880ad914857   grafana/grafana:latest   "sh -euc 'mkdir -p /…"   About an hour ago   Up About a minute   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   plg-grafana-1
dcc6c716cd69   grafana/loki:2.8.0       "/usr/bin/loki -conf…"   About an hour ago   Up About a minute   0.0.0.0:3100->3100/tcp, :::3100->3100/tcp   plg-loki-1

3. Test the effect of the log scheme

3.1. Test 1: Promtail monitors /var/logdirectory changes

Experiment purpose: test whether Promtail can monitor changes in the /var/log directory

experiment procedure:

1. Create a new file in the /var/log directory

# 输出'hello lpg'到/var/log/lpg.log 文件
echo 'hello lpg' > /var/log/lpg.log

2. Monitor the /var/log/lpg.log file on the console

This also confirms the role of Promtail: monitor /var/logchanges in a specific directory of the server (default is) and send the logs to Loki.

3. Continue to verify

• If /var/log/test.sqlI type something in, can it be monitored?
• If /var/log/fire/spring.logI type something in, can it be monitored?

3.2. Test 2: Grafana visual viewing log

Experiment purpose: View the logs monitored by Promtail through the visualization platform Grafana

experiment procedure:

1. Log in to Grafana, the account password is admin:admin. After successful login, you need to add Loki as the data source. Access address: http://192.168.56.123:3000/

   

2. A Loki data source is added by default after running the docker-compose.yml script.

   

3. Check out the data sources added by default. Afterwards, you can also set your Loki access address, click Save&testSave and Test, and a green prompt message will appear to indicate that the setting is successful.

   http://loki:3100 in the picture below is the network installed by loki script. You can learn more about it.docker network

   

4. Next, Exploreselect Loki and enter the query expression (Loki query) {filename="/var/log/lpg.log"}to see 测试1the logs.

   

3.3. Test 3: It is foreseeable that the logs of the deployed Spring Boot program can also be viewed by Grafana

Experiment purpose: Distributed monitoring of Spring Boot logs

experiment procedure:

In the experiment, a new Spring Boot application was created. In the Spring Boot configuration file application.yml, the logger.path=/var/logoutput directory of the log was specified as /var/log. The default output name of the Spring Boot log file is spring.log, that is to say The /var/log/spring.log file is used as the output log file. This file complies with the rules of Promtail configuration /var/log/*.log, so it should be monitored. Finally, check whether the log can be queried on the visualization platform Grafana.

Proceed as follows:

1. Partial configuration content of application.yml of Spring Boot application

# 配置日志文件的输出目的地
logging:
  path: /var/log

2. Upload the prepared jar package of the Spring Boot application to the Promtail server

3. Start the application and observe whether /var/log/spring.log has content

java -jar fire-tiny-loki-1.0-SNAPSHOT.jar
tail -f /var/log/spring.log

4. Search logs on the visualization platform Grafana

Log in to http://loki:3100, then Exploreselect Loki and enter the query expression (Loki query) {filename="/var/log/spring.log"}to see 测试1the logs

**Summary:** Each server needs to deploy a Promtail service. The service monitors a specific log directory. When the directory changes, the content is sent to Loki for log storage. Finally, the logs of all microservices can be seen on the Grafana visualization platform. information.

3.4. Pitfall records

The author was really deceived and recorded the process of being deceived to help with the aftermath.

As mentioned in 实验1the introduction, Promtail monitors the /var/log directory by default, but some strange problems occurred. The author's /var/log/fire/spring.log, /var/log/test.sql and other files cannot be Found it through the Grafana visualization platform, why, o(╯□╰)o

Analysis process:

1. Some files can be monitored, such as /var/log/lpg.log mentioned above, but some files cannot be monitored. It is speculated that it may be a configuration problem.

2. Enter the Promtail container to view the configuration file /etc/promtail/config.yml. Part of the content is as follows:

# ...
# 部分内容
scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

3. Observe the configuration file __path__: /var/log/*log. According to this configuration, only files ending with .log in the /var/log directory will be monitored . Then the /var/log/fire/spring.log and /var/log/test.sql we configured will naturally will not be queried. Now that the problem is found, check the official website and __path__use it glob patterns. Then modify the configuration to /var/log/**/*.logtry. This configuration style is also similar to the ant-style style and is also commonly used in Spring. It should not be a big problem, so start modifying it.

4. Directly modify /etc/promtail/config.ymlthe configuration content of the container, but the Promtail container does not have editing commands such as vi installed by default, but it is okay. We use the most original echocommands of Linux to overwrite /etc/promtail/config.ymlthe file (mainly because of laziness, this is the simplest)

# 1、先进入容器
docker exec -it lpg-promtail-1 /bin/bash
# 2、cat查看内容
cat /etc/promtail/config.yml
# 3、把内容复制到编辑器进行编辑
# 只改动一行，把/var/log/*.log改为/var/log/**/*.log
# 4、把编辑后的内容输出了...处开始覆写
echo '...' > /etc/promtail/config.yml
# 5、最后退出promtail容器

Restart 3 containers

docker-compose down
docker-compose up

5. After restarting, create several new files and access them through http://192.168.56.123:3100

⚠️⚠️⚠️Pay special attention to ⚠️⚠️⚠️, the filename in the figure below refers specifically to the promtail configuration __path__: /var/log/**/*.log, so /app/logs/fire/spring.logyou can’t access it if you use something like this. After all, the host has a relationship with the visualization tool Grafana, and only Promtail has a relationship with Grafana!

6. Modify the Promtail configuration file by mounting the host and the container (choose one of steps 5 and 6)

Modify the configuration file on the host, and indirectly modify the Promtail configuration file by mounting the host and the container. The specific steps are as follows:

• Stop docker-compose and delete the original 3 docker containers

  # 停止容器
  docker-compose down
  # 删除容器
  docker rm lpg-grafana-1;
  docker rm lpg-loki-1;
  docker rm lpg-promtail-1;
  

• /app/etc/promtail/config.ymlCreate a new configuration file on the host

  Copy the config.yml file in the appendix

  # 创建目录
  mkdir -p /app/etc/promtail
  # 复制附录内容填写到...位置
  echo '...' > /app/etc/promtail/config.yml
  

  __path__Just fine-tune the file

  # ...
  # 部分内容
  scrape_configs:
  - job_name: system
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        __path__: /var/log/**/*.log
  

• Modify part of the docker-compose.yml script as follows:

  # 创建目录
  mkdir -p /app/logs
  

    # 日志收集器
    promtail:
      image: grafana/promtail
      container_name: lpg-promtail
      volumes:
        # 把主机的/app/logs挂载到容器的/var/log
        - /app/logs/:/var/log/
        - /app/etc/promtail:/etc/promtail/
      # 这里指的是容器内部的/etc/promtail/promtail.yml文件
      command: -config.file=/etc/promtail/promtail.yml
  

• Something ready to restart docker-compose

  docker-compose up
  

• After restarting, it is no problem to create several new files and access them through http://192.168.56.123:3100

4. Introduction to the official entry case

Preface: Please prepare the Docker compose environment for Linux

In the previous part of this article, the simple introduction to LPG did not introduce some other components, but in the official entry case introduction, the system was larger and some other components were introduced. Let's experience it together! Official Getting starting address

This guide helps readers create and use a simple Loki cluster. This cluster is intended for testing, development, and evaluation; it will not meet most production requirements.

Introduction to the experimental process:

1. The test environment runs the flog application to generate log lines.

2. Promtail is the agent (or client) of the test environment, which captures the log lines and pushes them to the Loki cluster through the gateway.

In a typical environment, the log-generating application and agent run together,

3. Grafana provides a way to query the logs stored in Loki and visualize the query results.

Architecture diagram:

The following is a brief description of the above figure:

• flog is a developed test log generator that can generate logs in some common (such as Apache, Nginx, RFC3164 or Json) formats

• Promtail monitors the generated logs and pushes them to the gateway (i.e. nginx) interface.

  flog and Promtail are framed with dotted lines and are understood as clients.

• gateway is actually nginx to implement request forwarding

  1. Forward Promtail’s request to push logs

  2. Forward Grafana’s request to check logs

• Loki write component is responsible for writing logs pushed by the gateway

• Loki read component is responsible for reading logs stored in MiniO

• Grafana is a visualization platform for browser access logs

Official entry-level case experiment steps:

4.1. Get the test environment

1. Prepare a separate directory

mkdir evaluate-loki
cd evaluate-loki

Note: The log generated by flog after starting the container exists in the hidden directory of the current folder (evaluate-loki) .data.

2. Download loki-config.yaml, promtail-local-config.yaml, and docker-compose.yaml3 scripts

wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/loki-config.yaml -O loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/promtail-local-config.yaml -O promtail-local-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/main/examples/getting-started/docker-compose.yaml -O docker-compose.yaml

Because the website is inaccessible by default abroad, the author provided three scripts that I downloaded for direct use. Readers can also use the reference blog to try to solve the problem of inaccessibility of raw.githubusercontent.com . The complete script and instructions are also provided in the appendix at the end of this article.

4.2. Deployment environment

Enter the current directory of evaluate-loki and use the following command to start it in the background

docker-compose up

After startup, you can see log output at intervals on the console. This is the output log of flog in action. Each json log is pushed to the interface by Promtail through the gateway /loki/api/v1/push.

Visit http://192.168.56.123:3101/ready to see if loki read is ready.

Visit http://192.168.56.123:3102/ready to see if loki write is ready.

Check whether all the docker containers defined in the docker-compose.yaml script are started. If they are all started, there should be no big problem. You can see that a total of 7 services are started.

4.3. Testing with Grafana

Log in to http://192.168.56.123:3000. A datasource has been configured by default. Next, click to Exploreselect Loki, enter the query expression (Loki query) {container="evaluate-loki-flog-1"}, and click Query.

Click Query repeatedly to view the latest logs generated by flog. One log per second is completely consistent with the configuration of flog. The official website also provides many other query expressions that you can try on your own.

{container="evaluate-loki-flog-1"} |= "GET"
{container="evaluate-loki-flog-1"} |= "POST"
{container="evaluate-loki-flog-1"} | json | status="401"
{container="evaluate-loki-flog-1"} != "401"

4. Appendix

4.1. docker-compose script file

There are three services defined in the script file, namely loki, promtail, and grafana. It is necessary to understand the script file!

The contents of the original docker-compose file are as follows (remarks added):

version: "3"

# 创建docker容器的网络，方便互通
networks:
  loki:

services:
  loki:
    image: grafana/loki:2.8.0
    ports:
      - "3100:3100"
    # loki执行的命令是容器中的/etc/loki/local-config.yaml来启动loki
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.8.0
    # 执行主机与docker容器的文件挂载关系
    volumes:
      - /var/log:/var/log
    # promtail执行的命令是容器中的/etc/promtail/config.yml来启动promtail
    command: -config.file=/etc/promtail/config.yml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    # grafana启动所执行的脚本
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy 
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    networks:
      - loki

4.2. local-config.yaml file

Use docker container /etc/loki/local-config.yamlto start loki. The contents of the original file taken out of the loki container are as follows (notes added):

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

4.3. config.yml file

Use docker container /etc/promtail/config.ymlto start promtail. The contents of the original file taken out of the promtail container are as follows (remarks added):

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*.log

• erverProperties configure Promtail's behavior as an HTTP server.
• positionsThe property configures the location where Promtail saves the file, indicating how far it has read the file. It is required when Promtail restarts, allowing it to continue reading logs from where it left off.
• scrape_configsProperties configure how Promtail crawls logs from a range of targets using the specified discovery method.
  • static_configsFetch log static target configuration, static configuration allows specifying a target list and tag set
  • labelDefines a log file to crawl and an optional set of additional tags to apply to the file log stream defined by __path__.

Detailed configuration reference:

• Promtail configuration file description: https://cloud.tencent.com/developer/article/1824988

• Promtail official documentation: https://grafana.com/docs/loki/latest/clients/promtail/configuration/

4.4. Official entry case script

4.4.1. docker-compose.yaml

This script is the build script of docker-compose. Here is a brief introduction:

1. Define several services in the architecture diagram (such as: flog, nginx, promtail, loki read, loki write, minio, grafana)

2. The startup of each service is basically started through commandor shscript.

• flog, loki read, loki write, promtail are through command method
• nginx, minio, grafana are through sh scripts

3. There are dependencies between services

• For example, loki read and loki write all depend on minio, and minio is the last one to provide storage services.

4. The downloaded and configuration files volumesare used by mounting the service.loki-config.yamlpromtail-local-config.yaml

---
version: "3"

networks:
  loki:

services:
  read:
    image: grafana/loki:2.8.3
    command: "-config.file=/etc/loki/config.yaml -target=read"
    ports:
      - 3101:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    depends_on:
      - minio
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks: &loki-dns
      loki:
        aliases:
          - loki

  write:
    image: grafana/loki:2.8.3
    command: "-config.file=/etc/loki/config.yaml -target=write"
    ports:
      - 3102:3100
      - 7946
      - 9095
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    depends_on:
      - minio
    networks:
      <<: *loki-dns

  promtail:
    image: grafana/promtail:2.8.3
    volumes:
      - ./promtail-local-config.yaml:/etc/promtail/config.yaml:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yaml
    depends_on:
      - gateway
    networks:
      - loki

  minio:
    image: minio/minio
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /data/loki-data && \
        mkdir -p /data/loki-ruler && \
        minio server /data
    environment:
      - MINIO_ROOT_USER=loki
      - MINIO_ROOT_PASSWORD=supersecret
      - MINIO_PROMETHEUS_AUTH_TYPE=public
      - MINIO_UPDATE=off
    ports:
      - 9000
    volumes:
      - ./.data/minio:/data
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 15s
      timeout: 20s
      retries: 5
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - gateway
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: Loki
            type: loki
            access: proxy
            url: http://gateway:3100
            jsonData:
              httpHeaderName1: "X-Scope-OrgID"
            secureJsonData:
              httpHeaderValue1: "tenant1"
        EOF
        /run.sh
    ports:
      - "3000:3000"
    healthcheck:
      test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

  gateway:
    image: nginx:latest
    depends_on:
      - read
      - write
    entrypoint:
      - sh
      - -euc
      - |
        cat <<EOF > /etc/nginx/nginx.conf
        user  nginx;
        worker_processes  5;  ## Default: 1

        events {
    
    
          worker_connections   1000;
        }

        http {
    
    
          resolver 127.0.0.11;

          server {
    
    
            listen             3100;

            location = / {
    
    
              return 200 'OK';
              auth_basic off;
            }

            location = /api/prom/push {
    
    
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /api/prom/tail {
    
    
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /api/prom/.* {
    
    
              proxy_pass       http://read:3100\$$request_uri;
            }

            location = /loki/api/v1/push {
    
    
              proxy_pass       http://write:3100\$$request_uri;
            }

            location = /loki/api/v1/tail {
    
    
              proxy_pass       http://read:3100\$$request_uri;
              proxy_set_header Upgrade \$$http_upgrade;
              proxy_set_header Connection "upgrade";
            }

            location ~ /loki/api/.* {
    
    
              proxy_pass       http://read:3100\$$request_uri;
            }
          }
        }
        EOF
        /docker-entrypoint.sh nginx -g "daemon off;"
    ports:
      - "3100:3100"
    healthcheck:
      test: ["CMD", "service", "nginx", "status"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - loki

  flog:
    image: mingrammer/flog
    command: -f json -n 1000 -l
    networks:
      - loki

4.4.2. loki-config.yaml

---
server:
  http_listen_port: 3100
memberlist:
  join_members:
    - loki:7946
schema_config:
  configs:
    - from: 2021-08-01
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
common:
  path_prefix: /loki
  replication_factor: 1
  storage:
    s3:
      endpoint: minio:9000
      insecure: true
      bucketnames: loki-data
      access_key_id: loki
      secret_access_key: supersecret
      s3forcepathstyle: true
  ring:
    kvstore:
      store: memberlist
ruler:
  storage:
    s3:
      bucketnames: loki-ruler

4.4.3. promtail-local-config.yaml

---
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://gateway:3100/loki/api/v1/push
    tenant_id: tenant1

scrape_configs:
  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'