basic concept

KMS, Key Management Service, is the key management service. In the K8S cluster, the encryption of Secret and Configmap is enabled in the form of drivers and plug-ins. to protect sensitive data,

Drivers and plug-ins require users to customize and implement their own KMS plug-ins according to requirements. The plug-in can be a gRPC server or a KMS plug-in provided by a cloud service provider.

The KMS service demonstrated in this article is the KMS encryption service in JD Cloud Ship.

At present, KMS is divided into V1 and V2. This article demonstrates based on V1.

architecture

You can use kms encryption internally to implement your own encryption algorithm, even the national secret algorithm.

When a user creates a new secret resource, kube-apiserver will call kms-plugin through gRPC, and kms-plugin communicates with the encryption server to encrypt data.

At this time, if the original data in etcd is obtained directly, the content is ciphertext data.

When the user obtains the content of the secret resource, the kube-apiserver will call the kms-plugin through gRPC, and the kms-plugin communicates with the encryption server to decrypt the data and display the plaintext to the user

Steps

A set of running Kubernetes cluster services is required. If there are multiple master nodes, they need to be configured at the same time.

new directory

/etc/kubernetes/kms/jdcloud

New EncryptionConfiguration

This configuration is the basic encryption configuration of kms, including encrypted resource objects, socket addresses, and so on.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets # 这里表示，只加密secret
    providers:
    - kms:
        name: myKmsPlugin
        endpoint: unix:///var/run/k8s-kms-plugin/kms-plugin.sock # 如果不以pod（jdcloud-kms-plugin.yaml）启动，需要sock文件放到master节点。
        cachesize: 100
        timeout: 3s
    - identity: {}

The above content is saved in /etc/kubernetes/kms/jdcloud/apiserver-encryption.conf

New jdcloud kms plugin configuration

Uplink information configuration of kms server

{
  "AccessKey": "xxx", # 部署前，该参数需要预先知道，
  "SecretKey": "yyy", # 部署前，该参数需要预先知道。
  "KmsEndpoint": "kms.internal.cn-north-1.jdcloud-api.com", # 部署前，该参数需要预先知道。
  "KmsKeyId": "abcd", # 部署前，该参数需要预先知道。
  "KmsSchema": "http",
  "GRPCSocketPath": "/var/run/k8s-kms-plugin/kms-plugin.sock"
}

The above content is saved in /etc/kubernetes/kms/jdcloud/jdcloud-kms-plugin.json

Create a new jdcloud kms plugin service

The service is to start the socket service, communicate with the kms server connected according to the configuration, encrypt and decrypt data, and interact with the K8S APIServer through the socket service.

The pod needs to be started before the kube-apiserver is started, otherwise there may be a circular dependency with the apiserver.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: jdcloud-kms-plugin
    tier: control-plane
  name: jdcloud-kms-plugin-node-01
  namespace: kube-system
spec:
  containers:
  - command:
    - /k8s-kms-plugin
    - -f=/etc/kubernetes/kms/jdcloud/jdcloud-kms-plugin.json # 指定json
    image: hub-pub.jdcloud.com/k8s/jdcloudsec/k8s-kms-plugin:v1.0.1 
    imagePullPolicy: IfNotPresent
    name: jdcloud-kms-plugin
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/kms/jdcloud/jdcloud-kms-plugin.json # 注意路径
      name: jdcloud-kms-plugin-configfile
      readOnly: true
    - mountPath: /var/run/k8s-kms-plugin/
      name: k8s-kms-plugin-unixsock-directory
      readOnly: false
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/kms/jdcloud/jdcloud-kms-plugin.json # 注意路径
      type: File
    name: jdcloud-kms-plugin-configfile
  - hostPath:
      path: /var/run/k8s-kms-plugin/
      type: DirectoryOrCreate
    name: k8s-kms-plugin-unixsock-directory
status: {}

The above content is saved in /etc/kubernetes/manifests/jdcloud-kms-plugin.yaml

Modify the kube apiserver configuration

...
    - --encryption-provider-config=/etc/kubernetes/kms/jdcloud/apiserver-encryption.conf
    image: hub-pub.jdcloud.com/k8s/kube-apiserver:v1.19.9-109
    imagePullPolicy: IfNotPresent
    livenessProbe:
...
    - mountPath: /etc/kubernetes/kms/jdcloud/apiserver-encryption.conf
      name: apiserver-encryption-conf
      readOnly: true
    - mountPath: /var/run/k8s-kms-plugin/
      name: k8s-kms-plugin-unixsock-directory
      readOnly: false
...
  - hostPath:
      path: /etc/kubernetes/kms/jdcloud/apiserver-encryption.conf
      type: File
    name: apiserver-encryption-conf
  - hostPath:
      path: /var/run/k8s-kms-plugin/
      type: DirectoryOrCreate
    name: k8s-kms-plugin-unixsock-directory

Save after modification

verify

Create a Secret named secret1 in the default namespace:

kubectl create secret generic secret1 -n default --from-literal=mykey=mydata

Use the etcdctl command line to read the Secret from etcd:

etcdctl.sh get /kubernetes.io/secrets/default/secret1 [...] | hexdump -C

The result is encrypted data

Verify that the Secret was correctly decrypted when retrieved by the API server:

kubectl describe secret secret1 -n default

The result is plaintext, mykey: mydata

product capability

In the K8S cluster, JD.com has always paid more attention to the encryption of sensitive data, especially when Yunjian faces more and more customers in the financial industry. Encryption services are basically standard configurations in Yunjian.

After product capability polishing and internal implementation, KMS encryption service, K8S automated cluster and one-click configuration creation have achieved good productization capabilities in the cloud ship, which can be created along with the cluster and enable KMS encryption service with one click.

reference:

1. Use KMS driver for data encryption

Author: Wang Xiaofei, Jingdong Technology

Source: Reprinted by JD Cloud developer community, please indicate the source

Use JDOS KMS service in K8S cluster to securely encrypt sensitive data | JD Cloud technical team