Kubernetes YAML best practices and strategies

Kubernetes workloads are most commonly defined in YAML format files.

One of the problems with YAML is that it is difficult to describe the constraints or relationships between manifest files.
What if you want to check whether all the images deployed to the cluster have been extracted from the trusted registry?
How to prevent workloads without Pod security policies from being submitted to the cluster?
Integrated static checking can catch errors and violations of policy closer to the development life cycle.
And because the effectiveness and security of resource definitions are improved, you can trust that production workloads follow best practices.

The static inspection ecosystem of Kubernetes YAML files can be divided into the following categories:

API Verifier: This type of tool verifies the given YAML list against the Kubernetes API server.
Built-in checker: This type of tool is bundled with conscious checks for security, best practices, etc.
Custom validators: These tools allow custom checks to be written in multiple languages (such as python and Javascript).

In this article, you will learn six different tools:

Kubeval
Kube-score
Config-lint
Copper
Conftest
Polaris

Let's Go ~~~

Benchmark service

First deploy a benchmark service for later testing and comparison

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-echo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http-echo
  template:
    metadata:
      labels:
        app: http-echo
    spec:
      containers:
      - name: http-echo
        image: hashicorp/http-echo
        args: ["-text", "hello-world"]
        ports:
        - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: http-echo
spec:
  ports:
  - port: 5678
    protocol: TCP
    targetPort: 5678
  selector:
    app: http-echo

The deployment is completed and verified as follows:

[root@k8s-node001 Test]# kubectl  get po
NAME                            READY   STATUS    RESTARTS   AGE
http-echo-57dd74545-rtxzm       1/1     Running   0          65s
http-echo-57dd74545-trst7       1/1     Running   0          65s

[root@k8s-node001 Test]# kubectl  get svc
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
http-echo        ClusterIP   10.102.221.64   <none>        5678/TCP         70s

[root@k8s-node001 Test]# curl  10.102.221.64:5678
hello-world

The above YAML file can be deployed successfully, but does it follow best practices?

Let's start.

kubeval

The premise of kubeval is that any interaction with Kubernetes is done through its REST API.
Therefore, the API pattern can be used to verify whether a given YAML input conforms to the pattern.

Install kubeval

wget https://github.com/instrumenta/kubeval/releases/latest/download/kubeval-linux-amd64.tar.gz
tar xf kubeval-linux-amd64.tar.gz
cp kubeval /usr/local/bin

Now let's change base.yaml and delete

  selector:
    matchLabels:
      app: http-echo

Then use kubeval to check base.yaml

[root@k8s-node001 Test]# kubeval  base.yaml
WARN - base.yaml contains an invalid Deployment (http-echo) - selector: selector is required
PASS - base.yaml contains a valid Service (http-echo)

The output sees a WARN indicating that the selector is a required field,
then restore the selector, and check again

[root@k8s-node001 Test]# kubeval  base.yaml 
PASS - base.yaml contains a valid Deployment (http-echo)
PASS - base.yaml contains a valid Service (http-echo)

Check PASS

The advantage of tools like kubeval is that we can detect such errors early in the deployment cycle.
In addition, you don't need to visit the cluster to run the checks-they can be run offline.
By default, kubeval will validate resources based on the latest unpublished Kubernetes API pattern.
For more usage details, please refer to the official website

be a score

kube-score will analyze the YAML list you provide and score it against the cluster's built-in checks.
kube-score provides online and offline versions. If you are
lazy, use the online version.
First open https://kube-score.com/, and then paste the written YAML list in the input box. Here, the above text base.yaml will analyze

Kubernetes YAML best practices and strategies
The analysis results are as follows

Kubernetes YAML best practices and strategies

From the above, you can see the suggestions given for this file, such as resource restrictions, mirroring TAG, Pod network strategy, etc. Not bad, very useful tool. . .

Of course, kube-score is not scalable, and you cannot add or adjust strategies.
If you want to write custom checks to comply with organizational policies, you can use one of four tools: config-lint, copper, conftest or Polaris.

Config-lint

Config-lint is a tool for validating configuration files written in YAML, JSON, Terraform, CSV and Kubernetes manifests.

Install Config-lint

wget https://github.com/stelligent/config-lint/releases/download/v1.6.0/config-lint_Linux_x86_64.tar.gz
tar -zxf config-lint_Linux_x86_64.tar.gz
mv config-lint /usr/local/bin/

Config-lint does not perform built-in checks on the Kubernetes manifest. You must write your own rules to perform any verification.
The rules are written as YAML files, called rule sets, and have the following structure:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
   # list of rules

Suppose we want to check whether the image in the deployment is always extracted from a trusted repository (for example, kubeops.net/app:1.0).
The config-lint rules for implementing such checks are as follows:

- id: MY_DEPLOYMENT_IMAGE_TAG
  severity: FAILURE
  message: Deployment must use a valid image tag
  resource: Deployment
  assertions:
    - every:
        key: spec.template.spec.containers
        expressions:
          - key: image
            op: starts-with
            value: "kubeops.net/"

A complete rule set is as follows:

version: 1
description: Rules for Kubernetes spec files
type: Kubernetes
files:
  - "*.yaml"
rules:
  - id: DEPLOYMENT_IMAGE_REPOSITORY
    severity: FAILURE
    message: Deployment must use a valid image repository
    resource: Deployment
    assertions:
      - every:
          key: spec.template.spec.containers
          expressions:
            - key: image
              op: starts-with
              value: "kubeops.net/"

If you want to test the inspection, you can save the rule set as check_image_repo.yaml.
Then use config-lint to perform the check

[root@k8s-node001 Test]# config-lint -rules check_image_repo.yaml base.yaml
[
  {
    "AssertionMessage": "Every expression fails: And expression fails: image does not start with kubeops.net/",
    "Category": "",
    "CreatedAt": "2020-11-02T08:28:43Z",
    "Filename": "base.yaml",
    "LineNumber": 0,
    "ResourceID": "http-echo",
    "ResourceType": "Deployment",
    "RuleID": "DEPLOYMENT_IMAGE_REPOSITORY",
    "RuleMessage": "Deployment must use a valid image repository",
    "Status": "FAILURE"
  }
]

You can see Every expression fails, the test failed.
Now let's change the images address to image: kubeops.net/http-echo, and check again

[root@k8s-node001 Test]# config-lint -rules check_image_repo.yaml base.yaml
[]

If the output does not report an error, it means success.

Config-lint is a promising framework that allows you to use YAML DSL to write custom checks for Kubernetes YAML lists.
But what if you want to express more complex logic and inspections?
Does YAML have restrictions on this?
What if you can express these checks in a real programming language? Next look at Copper

Copper

Copper V2 is a framework that uses custom checks to verify the list, just like config-lint.
However, Copper does not use YAML definition checks.
Instead, the tests are written in JavaScript, and Copper provides a library containing some basic helpers to help read Kubernetes objects and report errors.

Install Copper

https://github.com/cloud66-oss/copper/releases/download/2.0.1/linux_amd64_2.0.1
mv linux_amd64_2.0.1 copper
chmod + x copper
mv copper /usr/local/bin/

Similar to config-lint, Copper does not provide built-in checks.
Let's customize a check to ensure that the deployment image tag must be non-latest.
check_image_repo.js

$$.forEach(function($){
    if ($.kind === 'Deployment') {
        $.spec.template.spec.containers.forEach(function(container) {
            var image = new DockerImage(container.image);
            if (image.tag === 'latest') {
                errors.add_error('no_latest',"latest is used in " + $.metadata.name, 1)
            }
        });
    }
});

Perform inspection

[root@k8s-node001 Test]# copper validate --in=base.yaml --validator=check_image_tag.js
Check no_latest failed with severity 1 due to latest is used in http-echo
Validation failed

Now amended toimage: kubeops.net/http-echo:v1.0.0

[root@k8s-node001 Test]# copper validate --in=base.yaml --validator=check_image_tag.js
Validation successful

For more usage, see

Conftest

Conftest is a testing framework for configuration data that can be used to check and verify the Kubernetes inventory.
The test is written using the dedicated query language Rego.

Install Conftest

wget https://github.com/open-policy-agent/conftest/releases/download/v0.21.0/conftest_0.21.0_Linux_x86_64.tar.gz
tar -xzf conftest_0.21.0_Linux_x86_64.tar.gz
mv conftest /usr/local/bin

Similar to config-lint and copper, conftest does not have any built-in checks.

First create a new directory conftest-checks and a file named check_image_registry.rego with the following content:

package main

deny[msg] {

  input.kind == "Deployment"
  image := input.spec.template.spec.containers[_].image
  not startswith(image, "kubeops.net/")
  msg := sprintf("image '%v' doesn't come from kubeops.net repository", [image])
}

First modify base.yaml and image: docker.io/http-echo
use conftest to perform detection

[root@k8s-node001 Test]# conftest test --policy ./conftest-checks base.yaml 
FAIL - base.yaml - image 'docker.io/http-echo:v1.0.0' doesn't come from kubeops.net repository

2 tests, 1 passed, 0 warnings, 1 failure, 0 exceptions

Modify it to base.yaml again,image: kubeops.net/http-echo

[root@k8s-node001 Test]# conftest test --policy ./conftest-checks base.yaml 

2 tests, 2 passed, 0 warnings, 0 failures, 0 exceptions

For more usage, see

Polaris

The last tool is Polaris. Polaris can be installed inside the cluster or used as a command line tool to statically analyze the Kubernetes list.
When run as a command line tool, it contains multiple built-in checks covering aspects such as security and best practices, similar to kube-score.
In addition, you can use it to write custom checks similar to config-lint, copper and conftest.
In other words, Polaris combines the advantages of two categories: built-in and custom checkers.

Install Polaris, here only install the command line mode

wget https://github.com/FairwindsOps/polaris/releases/download/1.2.1/polaris_1.2.1_linux_amd64.tar.gz
tar -zxf polaris_1.2.1_linux_amd64.tar.gz
 mv polaris /usr/local/bin/

After the installation is complete, you can use Polaris to check base.yaml
[root@k8s-node001 Test]# polaris audit --audit-path base.yaml The
result is as follows, there is a lot of information, here only part of the information is intercepted, you can read it carefully Look at the results of the analysis.

  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "base.yaml",
  "DisplayName": "base.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 1,
    "Namespaces": 0,
    "Controllers": 1
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {
          "hostIPCSet": {
            "ID": "hostIPCSet",
            "Message": "Host IPC is not configured",
            "Success": true,
            "Severity": "danger",
            "Category": "Security"
..............
              "tagNotSpecified": {
                "ID": "tagNotSpecified",
                "Message": "Image tag is specified",
                "Success": true,
                "Severity": "danger",
                "Category": "Images"
              }
            }
          }
        ]
      },
      "CreatedTime": "0001-01-01T00:00:00Z"
    }
  ]
}

In addition, you can output only the score

[root@k8s-node001 Test]# polaris audit --audit-path base.yaml --format score
66

The following YAML code snippet defines a new check called checkImageRepo:
config_with_custom_check.yaml

checks:
  checkImageRepo: danger

customChecks:
  checkImageRepo:
    successMessage: Image registry is valid
    failureMessage: Image registry is not valid
    category: Images
    target: Container
    schema:
      '$schema': http://json-schema.org/draft-07/schema
      type: object
      properties:
        image:
          type: string
          pattern: ^kubeops.net/.+$

Now the image of base.yaml is: image: docker.io/http-echo:v1.0.0
Let’s use custom rules to perform checks

[root@k8s-node001 Test]# polaris audit --config config_with_custom_check.yaml --audit-path base.yaml
{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "base.yaml",
  "DisplayName": "base.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 1,
    "Namespaces": 0,
    "Controllers": 1
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {},
        "ContainerResults": [
          {
            "Name": "http-echo",
            "Results": {
              "checkImageRepo": {
                "ID": "checkImageRepo",
                "Message": "Image registry is not valid",
                "Success": false,
                "Severity": "danger",
                "Category": "Images"
              }
            }
          }
        ]
      },
      "CreatedTime": "0001-01-01T00:00:00Z"
    }
  ]
}

The result shows "Message": "Image registry is not valid", "Success": false,
then modify the image of base.yaml to: image: kubeops.net/http-echo:v1.0.0
perform the check again

[root@k8s-node001 Test]# polaris audit --config config_with_custom_check.yaml --audit-path base.yaml 
{
  "PolarisOutputVersion": "1.0",
  "AuditTime": "0001-01-01T00:00:00Z",
  "SourceType": "Path",
  "SourceName": "base.yaml",
  "DisplayName": "base.yaml",
  "ClusterInfo": {
    "Version": "unknown",
    "Nodes": 0,
    "Pods": 1,
    "Namespaces": 0,
    "Controllers": 1
  },
  "Results": [
    {
      "Name": "http-echo",
      "Namespace": "",
      "Kind": "Deployment",
      "Results": {},
      "PodResult": {
        "Name": "",
        "Results": {},
        "ContainerResults": [
          {
            "Name": "http-echo",
            "Results": {
              "checkImageRepo": {
                "ID": "checkImageRepo",
                "Message": "Image registry is valid",
                "Success": true,
                "Severity": "danger",
                "Category": "Images"
              }
            }
          }
        ]
      },
      "CreatedTime": "0001-01-01T00:00:00Z"
    }
  ]
}

From the output, you see "Message": "Image registry is valid", "Success": true, and the check passed. . .
For more usage, see

to sum up

Although there are many tools for verifying, scoring, and organizing Kubernetes YAML files, it is important to have a healthy model to design and perform checks.
For example, if you want to consider a Kubernetes manifest through a pipeline, kubeval may be the first step in the pipeline because it can verify that the object definition conforms to the Kubernetes API pattern. Once this check is successful, you can proceed with more detailed tests, such as standard best practices and custom strategies.
Kube-score and Polaris are better choices.
If you have complex requirements and want to customize the details of the inspection, you should consider using copper, config-lint and conftest.
Although both conftest and config-lint use more YAML to define custom validation rules, Copper allows access to a true programming language, which makes it quite attractive.
But should you use one of them and write all the checks from scratch? Or should I use Polaris and just write other custom checks?
It all depends on you, the best is the one that suits you. . .