webhook Preliminary admission (compiled articles)

• Open-source compiler admission webhook
  • Compile the sample webhook
  • Mirrored
    • Dockerfile
    • Production commands
  • k8s configuration
    • 启用 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook
  • Deployment webhook
    • prepare
    • Webhook server deployment
    • Deployment of service
    • In k8s enabled in webhook
      • 配置 ValidatingWebhookConfiguration
      • 配置 MutatingWebhookConfiguration
  • Problems encountered
  • Reference Documents

Open-source compiler admission webhook

This article reference code admission Number-WebHook-Example , K8S-admission Number-WebHook document

under the influence of admission webhook my simple summary, when a user requests arrive k8s apiserver, apiserver based on MutatingWebhookConfigurationand ValidatingWebhookConfigurationconfiguration, the first call MutatingWebhookConfigurationto modify the configuration files requested by the user, and finally calls ValidatingWebhookConfigurationto verify this modified configuration file is legitimate.

We can use the mechanisms of mutating, will add some special configuration automatically, without the user to worry about. At the same time you can write code to set their own rules in validating the see if the request is legitimate.

As it has been studied how to write the code webhook / how to deploy webhook, to use good webhook a few points to note:

• Webhook nature of a http server, therefore, need to implement such a server in code, calling for apiserver
• mutating webhook and validating webhook respectively, to achieve different functions, both of which are not directly linked, in essence, if you modify / verification needs, we need to write two http server
• Writing good webhook, we recommended k8s deployment deployment, and the ability to use the service to expose
• apiserver whether to call webhook, is k8s MutatingWebhookConfiguration/ValidatingWebhookConfigurationdefined
• Authentication and rights need to pay attention when apiserver and webhook communication control problems

Compile the sample webhook

https://github.com/cnych/admission-webhook-exampleIt will be mutating project, validating functions are placed in a http server, which of course also possible. But k8s MutatingWebhookConfiguration/ValidatingWebhookConfigurationcertainly we need are deployed.

[root@localhost lihao04]# cd /root/lihao04
[root@localhost lihao04]# git clone https://github.com/cnych/admission-webhook-example.git
[root@localhost lihao04]# cd admission-webhook-example/
[root@localhost github.com]# export GOPATH=/root/lihao04/go/
[root@localhost github.com]# mkdir -p /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# ln -s /root/lihao04/admission-webhook-example /root/lihao04/go/src/github.com/cnych/
[root@localhost github.com]# export http_proxy=http://jarvis:[email protected]:3128/
[root@localhost github.com]# export https_proxy=http://jarvis:[email protected]:3128/
[root@localhost github.com]# cd /root/lihao04/go/src/github.com/cnych/admission-webhook-example
[root@localhost admission-webhook-example]# dep ensure -v
...

[root@localhost admission-webhook-example]# export GO111MODULE=on
[root@localhost admission-webhook-example]# CGO_ENABLED=0 GOOS=linux go build -mod=vendor -a -installsuffix cgo -o admission-webhook-example
go: creating new go.mod: module github.com/cnych/admission-webhook-example
go: copying requirements from Gopkg.lock
go: converting Gopkg.lock: stat github.com/json-iterator/go@f2b4162afba35581b6d4a50d3b8f34e33c144682: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/google/gofuzz@24818f796faf91cd76ec7bddd72458fbced7a6c1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/api@12444147eb1150aa5c80d2aae532cbc5b7be73d0: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/opencontainers/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/golang/glog@23def4e6c14b4da8ac2ed8007337bc5eb5007998: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/reflect2@1df9eeb2bb81f327b96228865c5687bc2194af3f: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apimachinery@e386b2658ed20923da8cc9250e552f082899a1ee: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiserver@88d4601515c27f180f7efc8705e4cc18dc19100d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat golang.org/x/net@892bf7b0c6e2f93b51166bf3882e50277fa5afc6: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/gogo/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/spf13/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/docker/distribution@34c706e759240975178df82495f147559cc0edc1: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/modern-go/concurrent@bacd9c7ef1dd9b15be4a9909b8ac7a4e313eec94: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/apiextensions-apiserver@f584b16eb23bd2a3fd292a027d698d95db427c5d: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat github.com/ghodss/[email protected]: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat k8s.io/kubernetes@efe960cdc41ee7b18d408128dfb80babb5bc746a: repo version lookup disabled by -mod=vendor
go: converting Gopkg.lock: stat gopkg.in/[email protected]: repo version lookup disabled by -mod=vendor

Mirrored

The compiled code packaged into image, so that you can use to deploy k8s deployment

Dockerfile

Note that the modification FROM

FROM alpine:3.10.3

ADD admission-webhook-example /admission-webhook-example

ENTRYPOINT ["./admission-webhook-example"]

Production commands

[root@localhost admission-webhook-example]# docker build --no-cache -t docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v1 .
Sending build context to Docker daemon     43MB
Step 1/3 : FROM docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
3.10.3: Pulling from jarvis-image/alpine
89d9c30c1d48: Already exists
Digest: sha256:e4355b66995c96b4b468159fc5c7e3540fcef961189ca13fee877798649f531a
Status: Downloaded newer image for docker-registry.lihao04.virtual/jarvis-image/alpine:3.10.3
 ---> 965ea09ff2eb
Step 2/3 : ADD admission-webhook-example /admission-webhook-example
 ---> 2c65bd92673c
Step 3/3 : ENTRYPOINT ["./admission-webhook-example"]
 ---> Running in 41ac87194f86
Removing intermediate container 41ac87194f86
 ---> 7c57f014ab9a
Successfully built 7c57f014ab9a
Successfully tagged docker-registry.lihao04.virtual/admission-webhook-example:v1

k8s configuration

启用 MutatingAdmissionWebhook 和 ValidatingAdmissionWebhook

MutatingAdmissionWebhookAnd ValidatingAdmissionWebhookby default is not enabled, apiserver you want to call webhook, enable the ability to have relevant

[root@master ~]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
  enableServiceLinks: true

Because the enable-admission-plugins missing feature, we want to enable

# 修改 /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook

After modifying the configuration file to take effect immediately

[root@master manifests]# kubectl get pods kube-apiserver-master.lihao04.virtual -n kube-system -o yaml|grep enable
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
  enableServiceLinks: true

# 确实重启过
[root@master manifests]# kubectl get pod -n kube-system|grep api
kube-apiserver-master.lihao04.virtual            1/1     Running            0          54s

Deployment webhook

prepare

This server is for apiserver called, it's what we run code compiled admission-webhook-example, he uses a mirror of what we build docker-registry.lihao04.virtual/admission-webhook-example:v1

Before deployment, you need to create CertificateSigningRequest, secret and so on, this is to communicate with apiserver webhook do authentication

[root@master lihao04]# cd /root/lihao04/admission-webhook-example/deployment
[root@master lihao04]# ./webhook-create-signed-cert.sh

Webhook server deployment

[root@master deployment]# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: admission-webhook-example-deployment
  labels:
    app: admission-webhook-example
spec:
  replicas: 1
  selector:
    matchLabels:
      app: admission-webhook-example
  template:
    metadata:
      labels:
        app: admission-webhook-example
    spec:
      containers:
        - name: admission-webhook-example
          image: docker-registry.lihao04.virtual/jarvis-image/admission-webhook-example:v2
          #image: cnych/admission-webhook-example:v1
          args:
            - -tlsCertFile=/etc/webhook/certs/cert.pem
            - -tlsKeyFile=/etc/webhook/certs/key.pem
            - -alsologtostderr
            - -v=4
            - 2>&1
          volumeMounts:
            - name: webhook-certs
              mountPath: /etc/webhook/certs
              readOnly: true
      volumes:
        - name: webhook-certs
          secret:
            secretName: admission-webhook-example-certs

[root@master deployment]# kubectl apply -f deployment.yaml
deployment.apps/admission-webhook-example-deployment create

Deployment of service

Let the deployment (webhook server) can be apiserver access, you must set a service, attention, access to 443 service will visit 443 ports webhook deployment of the pod, and the pod is the start of service on port 443, which we can look admission-webhook-example source code.

[root@master deployment]# cat service.yaml
apiVersion: v1
kind: Service
metadata:
  name: admission-webhook-example-svc
  labels:
    app: admission-webhook-example
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: admission-webhook-example

[root@master deployment]#  kubectl create -f service.yaml

In k8s enabled in webhook

When webhook server deployed after, apiserver know how it is to access webhook server?

Therefore, you need to configure a cluster k8s, let webhook k8s perceived to be, and to be able to call at the appropriate time.

Webhook function because there are two parts

• modify
• verification

Prior modification, after verification, so that the two functions need to be configured independently;

配置 ValidatingWebhookConfiguration

Configuration is named: ValidatingWebhookConfiguration

[root@master deployment]# cat validatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validation-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: required-labels.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/validate"
      # 通过 kubectl config view --raw --flatten -o json|grep certificate-authority-data 获得
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    # 对 namespace 中打了 admission-webhook-example = enabled 的进行验证，其他 namespace 不验证
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

[root@master deployment]# kubectl apply -f validatingwebhook-ca-bundle.yaml
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook-example-cfg created

test

[root@master deployment]# cat sleep.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 3
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      containers:
      - name: sleep
        image: tutum/curl
        command: ["/bin/sleep","infinity"]
        imagePullPolicy: IfNotPresent

[root@master deployment]# kubectl apply -f sleep.yaml
Error from server (required labels are not set): error when creating "sleep.yaml": admission webhook "required-labels.qikqiak.com" denied the request: required labels are not set

It can be found, if not with specific content, being given, to the verification results

配置 MutatingWebhookConfiguration

Configuration is named: MutatingWebhookConfiguration

[root@master deployment]# cat mutatingwebhook-ca-bundle.yaml
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-example-cfg
  labels:
    app: admission-webhook-example
webhooks:
  - name: mutating-example.qikqiak.com
    clientConfig:
      service:
        name: admission-webhook-example-svc
        namespace: default
        path: "/mutate"
      caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1Ea3lOekEzTVRZd04xb1hEVEk1TURreU5EQTNNVFl3TjFvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUlFCmZnRDJva2RpK3BiK2U5VmVMRFJaamxQRG9UTGpwS1NVc05OaHVSbW9Xbmoyek9od3J5cGRmd2V3QTJtdlYyVGIKdDFKYTRHSENWRmNZeWVveFE2REw5SkVLWGZoOGxKQkZxS0JuNzc3RFlFWXl5b1hqWjE5Wk5NUUZzUnZhL2RtRgpPVGlrR3ZIdDZqS0xiQmhqVlRhNTlPbmhXZTFiRUlGR0I3SXNlcG8vRXEwSDhPMW53UlVONEVYRnBPMXhiQktxCk5saS9IQ0FLZ2pWRlQrWjdpUjY3QXlMbzFUUk5JRkl1VkhDU2xRaERjaU9xTE1OM1FLeWN5ZnY2VXFsNllSeW8KeW42eGFkb1JQM2RJOUlnNWpJQ3c0dUNZeEJlbW14YnNURkQ5Tm90YkdqSnBNeVhKbGhjcGM1aVlUVDR3WUJMUwpRakNkcWlJYmNzODNSWkhqTW1jQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFMeTN4ZWZFYmtUaUgxczVmbHlMVzZEb08wNUgKdWMrUW83RmdnSmlqZjA3eFNwVXpYVTRiVjhtSk55RlFJS1BiRzhHa3dRRFhiVzMvYXlZa2gyT3Z1Si9Bb2U5dAoxeGlFL3NMbTlGeTdlMTJRenZFcjlsanVpMzJWSjBtYkRpVm9Sd3FEMFh0R1JnZGhVeGltdU9PZzd0aXd1WUtkCkhBY0NDdUNReHRnWXpuQXdnVnJYVWpaSnRRV2RoNUpiVUwrZGp6ejYxSVdpTytiQSt2c0d3cjRoV09SVlk2K0sKcFNIMlhiL2JIaG5XOHBSREdsbEl2U2piZmlzY0d6SU1tUHN5end4ZG9sRXlqcnpCdGdLaW5pcjJvZmdJcEVHQwpuUzdZSDR3WE1kTkp4TEQ3U2JWdWpDbEwwQzhaUWFvamFkNUlkU2FZaldjSHFKelFmczRaKzQvRHN1OD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    rules:
      - operations: [ "CREATE" ]
        apiGroups: ["apps", ""]
        apiVersions: ["v1"]
        resources: ["deployments","services"]
    namespaceSelector:
      matchLabels:
        admission-webhook-example: enabled

[root@master deployment]# kubectl apply -f mutatingwebhook-ca-bundle.yaml
mutatingwebhookconfiguration.admissionregistration.k8s.io/mutating-webhook-example-cfg created

test

The same task sleep

[root@master deployment]# kubectl apply -f sleep.yaml
deployment.apps/sleep created

You can create out, and you can see, a lot more additional label, because the information is added, it also passed the validation

[root@master deployment]# kubectl describe deployment sleep
Name:                   sleep
Namespace:              default
CreationTimestamp:      Fri, 25 Oct 2019 19:17:51 +0800
Labels:                 app.kubernetes.io/component=not_available
                        app.kubernetes.io/instance=not_available
                        app.kubernetes.io/managed-by=not_available
                        app.kubernetes.io/name=not_available
                        app.kubernetes.io/part-of=not_available
                        app.kubernetes.io/version=not_available
Annotations:            admission-webhook-example.qikqiak.com/status: mutated
                        deployment.kubernetes.io/revision: 1
Selector:               app=sleep
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=sleep
  Containers:
   sleep:
    Image:      tutum/curl
    Port:       <none>
    Host Port:  <none>
    Command:
      /bin/sleep
      infinity
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   sleep-674f75ff4d (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  57s   deployment-controller  Scaled up replica set sleep-674f75ff4d to 3

Problems encountered

1. webhook server authentication does not work, we found to be bad certificate

[root@master deployment]# kubectl log admission-webhook-example-deployment-5594df959f-6rjql
log is DEPRECATED and will be removed in a future version. Use logs instead.
I1025 09:32:21.348823       1 main.go:50] Server started
2019/10/25 09:33:01 http: TLS handshake error from 10.244.0.0:10131: remote error: tls: bad certificate

Found, reference documentation, the following steps have problems

cat ./deployment/validatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/validatingwebhook-ca-bundle.yaml

Reference Documents

• https://www.qikqiak.com/post/k8s-admission-webhook/ (Chinese)
• https://banzaicloud.com/blog/k8s-admission-webhooks/ (English)