Logging with Fluentd

这个task展示如何配置Istio去创建定制日志记录并发送它们到Fluentd 后台进程。Fluentd 是一个开源日志收集器，支持多种数据输出类型（ data outputs ），并有一个可插拔的架构。一个流行的日志后端是 Elasticsearch,，然后使用 Kibana 作为查看器。完成这个task后，一个新的日志流将会发送日志到 Fluentd / Elasticsearch / Kibana 栈中。

这个task中使用 Bookinfo 示例。

Before you begin

在集群中安装Istio并部署一个应用。这个task假设Mixer以默认配置安装(--configDefaultNamespace=istio-system)。如果你使用不同的值，在这个task中更新配置和命令中对应值。

Setup Fluentd

在你的集群中，你可能已经有一个Fluentd 后台进程正在运行，像 here and here所描述的插件，或者你的集群提供的特定内容。这可以通过配置发送日志到Elasticsearch 系统或日志提供者。

你可能使用这些Fluentd 后台进程，或其他你建立的Fluentd 后台进程，只要它们监听转发日志，并且Mixer能够和它们建立连接。为了Mixer能够连接一个运行的Fluentd 后台进程，你可能需要添加一个Fluentd 的 service 。监听转发日志的 Fluentd 配置如下：

<source>
  type forward
</source>

将Mixer连接到所有可能的Fluentd配置的完整细节不在本task讨论范围内。

Example Fluentd，Elasticsearch，Kibana Stack

为了达到这个task的目的，你可能要部署提供的示例栈。这个栈包括 Fluentd, Elasticsearch, and Kibana ，它们都位于一个名为logging 的新命名空间，是一组不在生产环境中的 Services and Deployments 。

保存下面内容到 logging-stack.yaml.

# Logging Namespace. All below are a part of this namespace.
apiVersion: v1
kind: Namespace
metadata:
  name: logging
---
# Elasticsearch Service
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    app: elasticsearch
---
# Elasticsearch Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.1.1
        name: elasticsearch
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: discovery.type
            value: single-node
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: elasticsearch
          mountPath: /data
      volumes:
      - name: elasticsearch
        emptyDir: {}
---
# Fluentd Service
apiVersion: v1
kind: Service
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    app: fluentd-es
spec:
  ports:
  - name: fluentd-tcp
    port: 24224
    protocol: TCP
    targetPort: 24224
  - name: fluentd-udp
    port: 24224
    protocol: UDP
    targetPort: 24224
  selector:
    app: fluentd-es
---
# Fluentd Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: fluentd-es
  namespace: logging
  labels:
    app: fluentd-es
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  template:
    metadata:
      labels:
        app: fluentd-es
    spec:
      containers:
      - name: fluentd-es
        image: gcr.io/google-containers/fluentd-elasticsearch:v2.0.1
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: config-volume
          mountPath: /etc/fluent/config.d
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-es-config
---
# Fluentd ConfigMap, contains config files.
kind: ConfigMap
apiVersion: v1
data:
  forward.input.conf: |-
    # Takes the messages sent over TCP
    <source>
      type forward
    </source>
  output.conf: |-
    <match **>
       type elasticsearch
       log_level info
       include_tag_key true
       host elasticsearch
       port 9200
       logstash_format true
       # Set the chunk limits.
       buffer_chunk_limit 2M
       buffer_queue_limit 8
       flush_interval 5s
       # Never wait longer than 5 minutes between retries.
       max_retry_wait 30
       # Disable the limit on the number of retries (retry forever).
       disable_retry_limit
       # Use multiple threads for processing.
       num_threads 2
    </match>
metadata:
  name: fluentd-es-config
  namespace: logging
---
# Kibana Service
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    app: kibana
---
# Kibana Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana-oss:6.1.1
        resources:
          # need more cpu upon initialization, therefore burstable class
          limits:
            cpu: 1000m
          requests:
            cpu: 100m
        env:
          - name: ELASTICSEARCH_URL
            value: http://elasticsearch:9200
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
---

新建资源：

kubectl apply -f logging-stack.yaml

你将看到如下内容：

namespace "logging" created
service "elasticsearch" created
deployment "elasticsearch" created
service "fluentd-es" created
deployment "fluentd-es" created
configmap "fluentd-es-config" created
service "kibana" created
deployment "kibana" created

Configure Istio

现在这有一个正在运行的Fluentd 后台进程，用一个新的日志类型来配置Istio，并发送这些日志到监听后台进程。新建一个YAML文件来保存Istio自动生成和收集日志流的配置：
保存如下内容到fluentd-istio.yaml:

# Configuration for logentry instances
apiVersion: "config.istio.io/v1alpha2"
kind: logentry
metadata:
  name: newlog
  namespace: istio-system
spec:
  severity: '"info"'
  timestamp: request.time
  variables:
    source: source.labels["app"] | source.service | "unknown"
    user: source.user | "unknown"
    destination: destination.labels["app"] | destination.service | "unknown"
    responseCode: response.code | 0
    responseSize: response.size | 0
    latency: response.duration | "0ms"
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a fluentd handler
apiVersion: "config.istio.io/v1alpha2"
kind: fluentd
metadata:
  name: handler
  namespace: istio-system
spec:
  address: "fluentd-es.logging:24224"
---
# Rule to send logentry instances to the fluentd handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: newlogtofluentd
  namespace: istio-system
spec:
  match: "true" # match for all requests
  actions:
   - handler: handler.fluentd
     instances:
     - newlog.logentry
---

新建资源：

istioctl create -f fluentd-istio.yaml

预期输出类似：

Created config logentry/istio-system/newlog at revision 22374
Created config fluentd/istio-system/handler at revision 22375
Created config rule/istio-system/newlogtofluentd at revision 22376

注意 address: "fluentd-es.logging:24224" 行在处理器配置中指定了在示例栈中我们建立的Fluentd 后台进程。

View the new logs

1，为示例应用发送流量。
对于 Bookinfo ，在你的浏览器中访问 http://$GATEWAY_URL/productpage 或执行如下命令：

curl http://$GATEWAY_URL/productpage

2.在k8s环境中，通过如下命令设置Kibana 的端口转发：

kubectl -n logging port-forward $(kubectl -n logging get pod -l app=kibana -o jsonpath='{.items[0].metadata.name}') 5601:5601

让这个命令运行，当能够访问 Kibana UI时，按 Ctrl-C 退出。

3.操作 Kibana UI 并点击右上角 “Set up index patterns”。

4.使用 * 作为索引模式，然后点击 “Next step.”。

5.选择 @timestamp 作为 Time Filter 列名，然后点击“Create index pattern.”。

6.现在在左菜单点击“Discover” ，然后开始发现生成的日志。

Cleanup

移除新的遥测配置：

istioctl delete -f fluentd-istio.yaml

移除示例 Fluentd, Elasticsearch, Kibana 栈：

kubectl delete -f logging-stack.yaml

如果你不打算探索接下来地任何课题，参考 Bookinfo cleanup 指南来关闭应用。