kubernetes -- helm charts 开发: 2、 编写daemonset

1 概要介绍

DaemonSet:
特点：
确保全部Node上运行一个Pod副本（单副本）
Node加入集群，则新增Pod；移除，则回收
删除Daemonset,删除创建的Pod

2编写

2.1 必须字段

apiVersion:
可用值： extensions/v1beta1
kind :
可用值：DaemonSet
metadata:
例如：name: nova-compute
.spec:
唯一必须的字段：
.spec.template
例如：
    metadata:
      labels:
{{ tuple $envAll .Chart.Name $daemonset | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
      annotations:
        configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}

.spec.template：
含义：是Pod模板。Pod有明确的格式。Pod中必须指明标签合重启策略，默认always。
具体参见：
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#pod-template

2.2 Pod Selector

.spec.selector：
表示Pod Selector.是对象，包含:
matchLabels:
matchEx[ressions:

终于发现不能运行的原因：
zabbix-damonset:
{{ tuple $envAll "zabbix_agent" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}

nova-daemonset:
{{ tuple $envAll $daemonset | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}

ceilometer现在写的是：
{{ tuple $envAll | include "helm-toolkit.snippets.kubernetes_upgrades_deployment" | indent 2 }}

如果指定spec.selector必须和.spec.template.metadata.labels
匹配，目前没有发现spec.selector

2.3 .spec.template.spec.nodeSelector

如果指定.spec.template.spec.nodeSelector，则Daemonset Controller将在能够匹配上Node Selector的Node上创建Pod,在能够匹配上的Node Affinity的Node上创建Pod。
目前nova:
      nodeSelector:
        {{ .Values.labels.agent.compute.node_selector_key }}: {{ .Values.labels.agent.compute.node_selector_value }}

没有affinity，所以会在所有Node上创建Pod

ceilometer:
      nodeSelector:
        {{ .Values.labels.compute.node_selector_key }}: {{ .Values.labels.compute.node_selector_value }}
      affinity:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_pod_anti_affinity" | indent 8 }}

3 如何调度Daemon Pod

Pod运行在哪个机器是由Kubernetes调度器进行选择的，Daemon Controller创建的Pod已经确定在哪个机器上。

4 与Daemon Pod通信

模式:
Push: Pod向其他Service发送更新
NodeIP和的端口：使用hostPort,可通过Node IP访问到Pod
客户端知道Node IP列表
DNS:创建相同Pod Selector的Headless Service.通过endpoints资源检索
Service:创建具有相同Pod Selector的Service,用该Service访问到随机Node上的daemon

5 更新DaemonSet

修改Node Label,DaemonSet向新匹配的Node添加Pod

6 init脚本

通过直接在Node上启动daemon进程。

7 一个ceilometer-compute的daemonset例子

7.1修改文件统计

values.yaml
daemonset-compute.yaml

7.2 values.yaml

manifest部分:
  daemonset_compute: true
  deployment_notification: true

lifecycle部分:
添加：
      daemonsets:
        pod_replacement_strategy: RollingUpdate
        compute:
          enabled: true
          min_ready_seconds: 0
          max_unavailable: 1

replicas部分：
删除,容易遗漏,因为daemonset没有副本数的概念，deployment才有
    compute: 3

7.3 daemonset-compute.yaml

判断部分：
{{- if .Values.manifests.daemonset_compute }}
appversion：
apiVersion: extensions/v1beta1
kind：
kind: DaemonSet
affinity：
删除（其他都没有）
daemonset模板
{{ tuple $envAll "compute" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}

7.4 最终的ceilometer-compute的daemonset样例

{{- if .Values.manifests.daemonset_compute }}
{{- $envAll := . }}
{{- $dependencies := .Values.dependencies.compute }}
{{- $mounts_ceilometer_compute := .Values.pod.mounts.ceilometer_compute.ceilometer_compute }}
{{- $mounts_ceilometer_compute_init := .Values.pod.mounts.ceilometer_compute.init_container }}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ceilometer-compute
spec:
{{ tuple $envAll "compute" | include "helm-toolkit.snippets.kubernetes_upgrades_daemonset" | indent 2 }}
  selector:
    matchLabels:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 6 }}
  template:
    metadata:
      labels:
{{ tuple $envAll "ceilometer" "compute" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
      annotations:
        configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}
        configmap-etc-hash: {{ tuple "configmap-etc.yaml" . | include "helm-toolkit.utils.hash" }}
    spec:
      hostNetwork: true
      hostPID: true
      dnsPolicy: ClusterFirstWithHostNet
      nodeSelector:
        {{ .Values.labels.compute.node_selector_key }}: {{ .Values.labels.compute.node_selector_value }}
      initContainers:
{{ tuple $envAll $dependencies $mounts_ceilometer_compute_init | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
      containers:
        - name: ceilometer-compute
          image: {{ .Values.images.tags.ceilometer_compute }}
          imagePullPolicy: {{ .Values.images.pull_policy }}
{{ tuple $envAll $envAll.Values.pod.resources.compute | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
          command:
            - /tmp/ceilometer-compute.sh
          volumeMounts:
            - name: pod-etc-ceilometer
              mountPath: /etc/ceilometer
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/ceilometer.conf
              subPath: ceilometer.conf
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/api_paste.ini
              subPath: api_paste.ini
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/policy.json
              subPath: policy.json
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/event_definitions.yaml
              subPath: event_definitions.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/event_pipeline.yaml
              subPath: event_pipeline.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/pipeline.yaml
              subPath: pipeline.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/gnocchi_resources.yaml
              subPath: gnocchi_resources.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/mapping.json
              subPath: mapping.json
              readOnly: true
            - name: ceilometer-bin
              mountPath: /tmp/ceilometer-compute.sh
              subPath: ceilometer-compute.sh
              readOnly: true
            - name: varlibnova
              mountPath: /var/lib/nova
            - name: varliblibvirt
              mountPath: /var/lib/libvirt
            - name: run
              mountPath: /run
            - name: cgroup
              mountPath: /sys/fs/cgroup
            - name: machine-id
              mountPath: /etc/machine-id
              readOnly: true
{{ if $mounts_ceilometer_compute.volumeMounts }}{{ toYaml $mounts_ceilometer_compute.volumeMounts | indent 12 }}{{ end }}
      volumes:
        - name: pod-etc-ceilometer
          emptyDir: {}
        - name: ceilometer-etc
          configMap:
            name: ceilometer-etc
            defaultMode: 0444
        - name: ceilometer-bin
          configMap:
            name: ceilometer-bin
            defaultMode: 0555
        - name: varlibnova
          hostPath:
            path: /var/lib/nova
        - name: varliblibvirt
          hostPath:
            path: /var/lib/libvirt
        - name: run
          hostPath:
            path: /run
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: machine-id
          hostPath:
            path: /etc/machine-id
{{ if $mounts_ceilometer_compute.volumes }}{{ toYaml $mounts_ceilometer_compute.volumes | indent 8 }}{{ end }}
{{- end }}

8 总结

daemonset相比较于deployment的最大区别就是当有新的节点加入进来的时候，daemonset会自动新建该daemonset的pod，这种场景往往适用于需要从每个新加入的节点获取某种信息。典型的例子就是: nova-compute, ceilometer-compute,因为当新增计算节点的时候，nova-compute和ceilometer-compute需要从新添加的计算节点上获取虚机信息。

参考:
[1] https://jimmysong.io/kubernetes-handbook/concepts/daemonset.html