Cloud Native Log Agent/Aggregator

Loggie is a lightweight, high-performance, cloud-native log collection Agent and transit processing Aggregator based on Golang. It supports multi-Pipeline and component hot-swapping, and provides: One-stack
log solution: supports log transit, filtering, Analysis, segmentation, log alarm, etc. Cloud-native log form:
fast and convenient container log collection method, native Kubernetes CRD dynamic configuration distribution Production - level features: based on long-term large-scale operation and maintenance experience, an all-round Based on Loggie, we can build a cloud-native scalable full-link log data platform.

concept

core data flow

● Source: Input source, indicating a specific input source. A Pipeline can have multiple different input sources. For example, file source indicates the log file collection source, and Kafka source is the input source for reading Kafka.
● Sink: output source, indicating a specific output source. A Pipeline can only be configured with one type of output source, but it can have multiple parallel instances. For example, Elasticsearch sink indicates that the log data will be sent to the remote Elasticsearch.
● Interceptor: Interceptor, which represents a log data processing component. Different interceptors can perform log parsing, segmentation, conversion, and current limiting according to the implementation. A Pipeline can have multiple Interceptors, and the data stream is processed in a chain through multiple Interceptors.
● Queue: Queue, currently there are memory queues.
● Pipeline: Pipeline, source/interceptor/queue/sink together form a Pipeline, different Pipeline data isolation.

Management and Control

● Discovery: Delivery of dynamic configuration, currently mainly for log configuration under Kubernetes, you can collect container logs by creating CRD instances such as LogConfig. In the future, it will support the docking of various configuration centers in the host form.
● Monitor EventBus: Each component can publish data to the EventBus Topic, and a specific Listener will monitor the Topic and perform consumption processing. It is mainly used to monitor the exposure or sending of data.
● Reloader: used for dynamic update of configuration.

characteristic

A new generation of cloud-native log collection and transmission methods

Rapid configuration and use based on CRD

Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRD, simply create some YAML files to build a series of data collection, transmission, processing, and sending pipelines.

apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
  name: tomcat
  namespace: default
spec:
  selector:
    type: pod
    labelSelector:
      app: tomcat
  pipeline:
    sources: |
      - type: file
        name: common
        paths:
          - stdout
          - /usr/local/tomcat/logs/*.log
    sinkRef: default
    interceptorRef: default

Supports multiple deployment architectures

● Agent: Deployed using DaemonSet, log files can be collected without mounting the volume on the business container
Sidecar: Support Loggie sidecar non-invasive automatic injection, no need to manually add to the Deployment/StatefulSet deployment template
Aggregator: Support Deployment independent deployment into the form of transit machine, It can receive the data sent by the aggregated Loggie Agent, or it can be used alone to consume and process various data sources

But regardless of the deployment architecture, Loggie still maintains a simple and intuitive internal design.

Lightweight and High Performance

Benchmark pressure test comparison
Configure Filebeat and Loggie to collect logs and send them to a Kafka topic without client-side compression. Kafka topic is configured with Partition 3.

In the case of ensuring that the Agent specification resources are sufficient, modify the number of collected files, send client concurrency (configure Filebeat worker and Loggie parallelism), and observe the respective CPU, Memory, and Pod network card sending rates.

Agent	File size	number of log files	send concurrency	CPU	MEM (rss)	NIC sending rate
Filebeat	3.2G	1	3	7.5~8.5c	63.8MiB	75.9MiB/s
Filebeat	3.2G	1	8	10c	65MiB	70MiB/s
Filebeat	3.2G	10	8	11c	65MiB	80MiB/s
Loggia	3.2G	1	3	2.1c	60MiB	120MiB/s
Loggia	3.2G	1	8	2.4c	68.7MiB	120MiB/s
Loggia	3.2G	10	8	3.5c	70MiB	210MiB/s

Adaptive sink concurrency
After enabling the sink concurrency configuration, Loggie can:
● Automatically adjust the parallel number of downstream data sending according to the actual situation of downstream data response, so as to maximize the performance of the downstream server without affecting its performance.
● When the upstream data collection is blocked, properly adjust the downstream data sending speed to alleviate the upstream blockage.

Lightweight streaming data analysis and monitoring

The log itself is a kind of general data that has nothing to do with platforms and systems. How to make better use of these data is the core competence that Loggie focuses on and develops.

Real-time parsing and conversion
Just configure the transformer interceptor and configure functional actions to realize:
● Parsing of various data formats (json, grok, regex, split…)
● Conversion of various fields (add, copy, move , set, del, fmt…)
● Supports conditional judgment and processing logic (if, else, return, dropEvent, ignoreError…)
can be used for:
● extract log level level from log, and drop DEBUG log
● mix and include json in log In the log form of plain and plain, you can judge the log in json format and process it
. According to the status code in the access log, add different topic words

Example:

interceptors:
  - type: transformer
    actions:
      - action: regex(body)
        pattern: (?<ip>\S+) (?<id>\S+) (?<u>\S+) (?<time>\[.*?\]) (?<url>\".*?\") (?<status>\S+) (?<size>\S+)
      - if: equal(status, 404)
        then:
          - action: add(topic, not_found)
          - action: return()
      - if: equal(status, 500)
        then:
          - action: dropEvent()

Detection, recognition and alarm
Help you quickly detect possible problems and abnormalities in the data, and issue an alarm in time.
Matching methods are supported:
● No data: no log data is generated within the configured time period
● Matching
Fuzzy matching
Regular matching
● Conditional judgment
Field comparison: equal/less/greater…

Support deployment forms:
● Data collection link detection: easy to use, no additional deployment required
● Independent link detection two forms: independent deployment of Aggregator, consumption of Kafka/Elasticsearch, etc. for data matching and alarming
can support custom webhook docking Various alarm channels

Business data aggregation and monitoring
In many cases, the indicator data metrics are not only exposed through the prometheus exporter, but the log data itself can also provide the source of the indicator. For example, by counting the access logs of the gateway, you can calculate the number of 5xx or 4xx statusCode within a period of time, aggregate the qps of a certain interface, calculate the total amount of transmitted bodies, and so on.
This feature is in beta. Example:

- type: aggregator
  interval: 1m
  select:
  # 算子：COUNT/COUNT-DISTINCT/SUM/AVG/MAX/MIN
  - {
    
    key: amount, operator: SUM, as: amount_total}
  - {
    
    key: quantity, operator: SUM, as: qty_total}
  groupby: ["city"]
  # 计算：根据字段中的值，再计算处理
  calculate:
  - {
    
    expression: " ${amount_total} / ${qty_total} ", as: avg_amount}

Fast troubleshooting and observability of the entire link

● Loggie provides configurable and rich data indicators, and the dashboard can be imported into grafana with one click.

● Use the Loggie terminal and help interface to quickly and conveniently troubleshoot the problems of Loggie itself and the problems in the data transmission process

FAQs

Loggie vs Filebeat/Fluentd/Logstash/Flume

Loggia	Filebeat	Fluentd	Logstash	Flume
Development language	Golang	Golang	Ruby	JRuby	Java
Multi-Pipeline	support	single queue	single queue	support	support
multiple output sources	support	Not supported, only one Output	configuration copy	support	support
Transit	support	not support	support	support	support
log alarm	support	not support	not support	not support	not support
Kubernetes container log collection	Support container stdout and container internal log files	Only container stdout is supported	Only container stdout is supported	not support	not support
Configuration delivery	Under Kubernetes, it can be configured through CRD, and the host scenario configuration center is gradually supporting it	manual configuration	manual configuration	manual configuration	manual configuration
monitor	Natively supports Prometheus metrics, and can be configured to output indicator log files separately, send metrics, etc.	The API interface is exposed, and an additional exporter is required to access Prometheus	Support API and Prometheus metrics	Need to use additional exporter	Need to use additional exporter
resource usage	Low	Low	generally	higher	higher

deploy

1. Prepare the Kubernetes environment

You can use an existing Kubernetes cluster, or deploy Kubernetes . It is recommended to use Kind to build a Kubernetes cluster locally .
The operations in this article need to be used locally:
kubectl ( download )
helm ( download )
Please ensure that there are kubectl and helm executable commands locally.

2. Deploy Loggie DaemonSet

You can view all published deployment charts on the installation page.
You can choose to download the chart and then deploy:

VERSION=v1.4.0
helm pull https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz && tar xvzf loggie-$VERSION.tgz

Try to modify the values.yaml in it. Please replace the above <VERSION>with the specific version number. Then deploy and install:
If the loggie namespace has been created in your environment, you can ignore the -nloggie and --create-namespace parameters. Of course, you can also use your own namespace, just replace loggie in it.

helm install loggie ./loggie -nloggie --create-namespace

Of course you can also deploy directly:

# 请将下面的<VERSION>替换成具体的版本号
helm install loggie -nloggie --create-namespace https://github.com/loggie-io/installation/releases/download/$VERSION/loggie-$VERSION.tgz

In order to experience the latest Fixes and features conveniently , we provide a mirror version of the main branch after each merge, which can be selected through <here> (https://hub.docker.com/r/loggieio/loggie/tags) . At the same time, you can add --set image=loggieio/loggie:vX.YZ to the helm install command to specify a specific Loggie image. ● Kubernetes version problem failed to install CRD crds/crds.yaml: unable to recognize “”: no matches for kind “CustomResourceDefinition” in version “apiextensions.k8s.io/v1” If you have similar problems during helm install, It means that your Kubernetes version is relatively low and does not support CRD of apiextensions.k8s.io/v1 version. Loggie temporarily retains the CRD of the v1beta1 version, please delete the v1beta1 version in the charts, and re-install.




rm loggie/crds/crds.yaml

3. Collect logs

Loggie defines the Kubernetes CRD LogConfig, and a LogConfig represents the log collection task for collecting a class of Pods.

3.1 Create collected Pods

We first create a Pod object for the collected logs. Next, the standard output stdout log of this Nginx Pod will be collected.

kubectl create deploy nginx --image=nginx

3.2 Define the output source Sink

Then we create an CRD Sinkinstance of Loggie definition, which indicates the backend for log sending. For the convenience of demonstration, here we send the log to the Loggie Agent's own log and print it.

cat << EOF | kubectl apply -f -
apiVersion: loggie.io/v1beta1
kind: Sink
metadata:
  name: default
spec:
  sink: |
    type: dev
    printEvents: true
EOF

You can kubectl get sinkview the created Sink by .

3.3 Define collection tasks

Loggie defines CRD LogConfig, which represents a log collection task. We create a LogConfig example as follows:

cat << EOF | kubectl apply -f -
apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    type: pod
    labelSelector:
      app: nginx
  pipeline:
    sources: |
      - type: file
        name: mylog
        paths:
        - stdout
    sinkRef: default
EOF

As you can see, sinkRef is used above to refer to the sink default CR just created. Of course, we can also use the sink field directly in Logconfig, examples are as follows:

cat << EOF | kubectl apply -f -
apiVersion: loggie.io/v1beta1
kind: LogConfig
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    type: pod
    labelSelector:
      app: nginx
  pipeline:
    sources: |
      - type: file
        name: mylog
        paths:
        - stdout
    sink: |
      type: dev
      printEvents: true
      codec:
        type: json
        pretty: true
EOF

After creation, we can use kubectl get lgc to view the created CRD instance. At the same time, we can also view LogConfig events through kubectl describe lgc nginx to get the latest status

Events:
  Type    Reason       Age   From                       Message
  ----    ------       ----  ----                       -------
  Normal  syncSuccess  52s   loggie/kind-control-plane  Sync type pod [nginx-6799fc88d8-5cb67] success

The above nginx LogConfig uses the spec.selector to match which Pod logs are collected. Here we use app: nginx to select the nginx Pod just created.
spec.pipeline indicates the Pipeline configuration of Loggie. We only collect the logs of the standard output of the container, so fill in stdout in paths.

4. View logs

First find the nginx pod node where it is located:

kubectl get po -owide -l app=nginx

Then we find the Loggie for that node:

kubectl -nloggie get po -owide |grep ${node}

able to pass:

kubectl -nloggie logs -f ${logge-pod}

View the log printed by Loggie, which shows the collected nginx standard output log.
A more comprehensive introduction to deployment: Deploying Loggie under Kubernetes
Best practices for log collection under Kubernetes: Log collection under Kubernetes

Use Loggie to collect container logs

How does Loggie collect container logs?

Due to the good scalability of Kubernetes, Kubernetes has designed a concept of custom resource CRD. Users can define CRD to express their desired state, and develop Controller with the help of some frameworks, and use Controller to turn our expectations into reality.
Based on this idea, which logs a service needs to collect and what kind of log configuration is required are the expectations of users, and this requires us to develop a log collection Controller to achieve it.
Therefore, users only need to fill in the logs of which Pods need to be collected in the CRD LogConfig we defined, and what is the log path in the Pod.
The core architecture is shown in the figure below:

Loggie will sense Pod and CRD events and dynamically update the configuration. At the same time, Loggie can find the corresponding file on the node to collect according to the Volume mounted on the log file path. In addition, according to the configuration, the Env/Annotation/Label on the Pod can be automatically added to the log as meta information.
At the same time, compared with the crude method of mounting the same path on all nodes for wildcard collection, it also solves the problem of being unable to fine-tune configuration and collect irrelevant logs for a single service.
Of course, the benefits are not limited to these. Loggie can be adapted and supported based on Kubernetes in terms of dynamic configuration delivery and monitoring indicators.

CRD Instructions for use

Loggie currently has the following CRDs:
LogConfig : namespace-level CRD, used to collect Pod container logs, which mainly fills in the collected source configuration, as well as the associated sink and interceptor.
ClusterLogConfig : Cluster-level CRD, which means collecting Pod container logs at the cluster level, collecting logs on Node nodes, and delivering general pipeline configurations for a Loggie cluster.
Sink : Indicates a sink backend, which needs to be associated in ClusterLogConfig/LogConfig.
Interceptor : Represents a group of interceptors that need to be associated in ClusterLogConfig/LogConfig.
The process architecture for using CRD is as follows:

If you do, you have a fifty percent chance. If you don't, you don't