Before building a K8S cluster in binary, we need to sort out the most boring point, which is the various certificates.
Official document reference : https://kubernetes.io/docs/setup/certificates/
How many certificates are there in total:
Calculate from Etcd first:
1. Etcd provides services to external parties and requires a set of etcd server certificate
2. To communicate between Etcd nodes, there must be a set of etcd peer certificates
3. Kube-APIserver must have a set of etcd client certificate to access Etcd
Calculate kubernetes again:
4. Kube-APIserver provides services externally, and a set of kube-apiserver server certificate is required
5. kube-scheduler, kube-controller-manager, kube-proxy, kubelet and other components that may be used need to access kube-APIserver and a set of kube-APIserver client certificates
6, kube-controller-manager wants to generate the service account of the service, and there must be a pair of certificates (CA certificates) used to sign the service account
7. When kubelet provides services externally, a set of kubelet server certificate is required
8. The kube-APIserver needs to access kubelet, and a set of kubelet client certificate is required
There are 8 sets in total, but we need to understand the meaning of "sets" here.
The certificates in the same set must be signed by the same CA, and the CAs that sign the certificates in different sets can be the same or different. For example, all etcd server certificates need to be signed by the same CA, and all etcd peer certificates also need to be signed by the same CA. An etcd server certificate and an etcd peer certificate can be signed by two CA institutions. not related. This counts as two sets of certificates.
Why the certificates in the same "set" must be signed by the same CA
The reason lies in the verification of these certificates. Because at the end to verify these certificates, usually only one Root CA can be specified . In this way, the certificate to be verified naturally needs to be signed by the private key corresponding to the same Root CA, otherwise it cannot pass the authentication.
In fact, K8S can be built using a set of certificates (all signed by a set of CAs), and it can be produced on the same basis, but the relationship between these certificates is clarified, and when the request is rejected due to certificate errors , It will not be impossible to start, and if the relationship between the certificates is not understood, the certificate is rashly replaced when maintaining or solving the problem, it will cause the entire system to be paralyzed.
TLS bootstrapping simplifies kubelet certificate production
Kubernetes 1.4 version introduced a set of APIs for signing certificates. The introduction of this set of APIs allows us to not prepare the certificates used by kubelet in advance.
Official website address: https://kubernetes.io/docs/tasks/tls/certificate-rotation/
The certificate used by each kubelet is unique, because it needs to bind its own IP address, so it is necessary to create a separate certificate for each kubelet. If the business volume is large, there will be many node nodes, so kubelet The number of kubelet has also increased, and the certificate creation of kubelet has become a very troublesome thing. Using TLS bootstrapping can save a lot of trouble.
How it works: When Kubelet is started for the first time, the same bootstrap token is used as a credential. This token has been set in advance to belong to the user group system: bootstrappers, and the permissions of this user group have also been restricted to only apply for certificates. After passing the authentication with this bootstrap token, kubelet applies for its own two sets of certificates (kubelet server, kube-apiserver client for kubelet). After the application is successful, it uses its own certificate for authentication, thus having the kubelet's due authority . In this way, the process of manually preparing a certificate for each kubelet is removed, and the kubelet certificate can also be automatically updated in rotation
Reference documents:
https://mritd.me/2018/01/07/kubernetes-tls-bootstrapping-note/
https://www.jianshu.com/p/bb973ab1029b
Why are kubelet certificates different
This is done for one audit and the other for security. Each kubelet is both a server (kube-apiserver needs to access kubelet) and a client (kubelet needs to access kube-apiserver), so there must be two sets of server and client certificates.
The server certificate needs to be bound to the server address. The address of each kubelet is different. Even if the domain name is bound, it is bound to a different domain name, so the server address is different
Client certificates should not be the same. After each kubelet's authentication certificate is bound to the IP of the machine where it is located, it can prevent a kubelet's authentication certificate from being leaked, allowing forged requests from another machine to pass verification.
In terms of security, if the bootstrap token used to sign the certificate is reserved on each node, after the bootstrap token is leaked, is it possible to sign the certificate at will? The safety hazard is very big. Therefore, after the kubelet is successfully started, the local bootstrap token needs to be deleted.
Official production certificate
Although multiple sets of certificates can be used, it is too complicated to maintain multiple sets of CAs. Here, one CA is used to sign all certificates.
Certificates to be prepared:
- admin.pem
- ca.-key.pem
- ca.pem
- admin-key.pem
- admin.pem
- kube-scheduler-key.pem
- kube-scheduler.pem
- kube-controller-manager-key.pem
- kube-controller-manager.pem
-
kube-proxy-key.pem
-
kube-proxy.pem
-
kubernetes-key.pem
-
kubernetes.pem
The components that use certificates are as follows:
-
etcd ： 使用 ca.pem kubernetes-key.pem kubernetes.pem
-
kube-apiserver 使用 使用 ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem
-
kubelet: use ca.pem
-
kube-proxy：使用 ca.pem kube-proxy-key.pem kube-proxy.pem
-
kubectl ca.pem admin-key.pem 、 admin.pem
-
kube-controller-manager：使用 ca-key.pem ca.pem kube-controller-manager-key.pem kube-controller-manager.pem
- kube-scheduler: 使用 kube-scheduler-key.pem kube-scheduler.pem
We use CFSSL to make certificates. It is an open source PKI tool developed by cloudflare. It is a complete CA service system that can sign and revoke certificates. It covers the entire life cycle of a certificate. Only its command line is used later. tool.
Note: Generally, the certificate in K8S only needs to be created once. When adding a new node to the cluster in the future, just copy the certificate in the /etc/kubernetes/ssl directory to the new node.
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
Create CA certificate
Create a certificate profile
vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"Kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
Field description:
ca-config.json: You can define multiple profiles, specify different expiration time, usage scenarios and other parameters; subsequently use a certain profile when signing the certificate;
signing: means that this certificate can sign other certificates; CA=TRUE in the generated ca.pem certificate;
server auth: indicates that the client can use the CA to verify the certificate provided by the server;
client auth: indicates that the server can use the CA to verify the certificate provided by the client;
expiry: expiration time
Create CA certificate signing request file
vim ca-csr.json
{
"CN" "kubernetes"
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
],
"ca": {
"expiry": "87600h"
}
}
Field description:
"CN": Common Name, kube-apiserver extracts this field from the certificate as the requested user name (User Name); the browser uses this field to verify whether the website is legitimate;
"O": Organization, kube-apiserver extracts this field from the certificate as the group to which the requesting user belongs
Generate CA certificate and private key
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls | grep ca
ca-config.json
ca.csr
ca-csr.json
ca-key.pem
ca.pem
Among them, ca-key.pem is the private key of ca, ca.csr is a signing request, and ca.pem is the CA certificate, which is the RootCA used by the kubernetes component later.
Create kubernetes certificate
Before creating this certificate, first plan the architecture
k8s-master1 10.211.55.11
k8s-master2 10.211.55.12
k8s-master3 10.211.55.13
etcd01 10.211.55.11
etcd02 10.211.55.12
etcd03 10.211.55.13
VIP 10.211.55.8
Create kubernetes certificate signing request file
I came from kubernetes-csr.json
{
"CN" "kubernetes"
"hosts": [
"127.0.0.1",
"10.211.55.11",
"10.211.55.12",
"10.211.55.13",
"10.211.55.8",
"10.0.0.1",
"k8s-master1",
"k8s-master2",
"k8s-master3",
"etcd01",
"etcd02",
"etcd03",
"Kubernetes"
"kube-api.wangdong.com",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
Field description:
If the hosts field is not empty, you need to specify a list of IP or domain names authorized to use the certificate.
Since this certificate is subsequently used by etcd cluster and kubernetes master cluster, fill in the IPs of etcd and master nodes, as well as the first IP of the service network. (Usually it is the first IP of the service-cluster-ip-range network segment specified by kube-apiserver, such as 10.0.0.1)
Three etcd, three master, the IP of the above physical node can also be replaced with the host name.
Generate kubernetes certificate and private key
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
ls | grep kubernetes
kubernetes.csr
kubernetes-csr.json
kubernetes-key.pem
kubernetes.pem
Create admin certificate
Create admin certificate signing request file
vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
Description:
Subsequent kube-apiserver uses RBAC to authorize client (such as kubelet, kube-proxy, Pod) requests;
kube-apiserver predefines some RoleBindings used by RBAC. For example, cluster-admin binds Group system:masters with Role cluster-admin. This Role grants the permission to call all APIs of kube-apiserver;
O specifies the group of the certificate as system:masters. When kubelet uses the certificate to access kube-apiserver, the certificate is signed by the CA, so the authentication is passed. At the same time, because the certificate user group is pre-authorized system:masters, it is granted access to all API permissions;
Note: This admin certificate is used to generate the kube config configuration file for administrators in the future. Now we generally recommend using RBAC to control the role and authority of kubernetes. Kubernetes uses the CN field in the certificate as the User and the O field as the Group.
For related permission authentication, please refer to the following article
https://mp.weixin.qq.com/s/XIkQdh5gnr-KJhuFHboNag
Generate admin certificate and private key
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
ls | grep admin
admin.csr
admin-csr.json
admin-key.pem
admin.pem
Create kube-proxy certificate
Create kube-proxy certificate signing request file
vim kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "k8s",
"OU": "System"
}
]
}
Description:
CN specifies the User of the certificate as system:kube-proxy;
kube-apiserver pre-defined RoleBinding system:node-proxier binds User system:kube-proxy with Role system:node-proxier, and this Role grants the permission to call kube-apiserver Proxy related APIs;
This certificate will only be used by kubectl as a client certificate, so the hosts field is empty
Generate kube-proxy certificate and private key
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
ls |grep kube-proxy
kube-proxy.csr
kube-proxy-csr.json
kube-proxy-key.pem
kube-proxy.pem
Create kube-controoler-manager certificate
Create kube-controoler-manager certificate signing request file
vim kube-controller-manager-csr.json
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"10.211.55.11",
"10.211.55.12",
"10.211.55.13",
"k8s-master1",
"k8s-master2",
"k8s-master3"
],
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-controller-manager",
"OU": "system"
}
]
}
Description:
The hosts list contains all kube-controller-manager node IPs;
CN is system:kube-controller-manager, O is system:kube-controller-manager, and the built-in ClusterRoleBindings system:kube-controller-manager in kubernetes gives kube-controller-manager the required permissions to work
Generate kube-controoller-manager certificate and private key
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
Create kube-scheduler certificate
Create kube-scheduler certificate signing request file
vim kube-scheduler-csr.json
{
"CN": "system:kube-scheduler",
"hosts": [
"127.0.0.1",
"10.211.55.11",
"10.211.55.12",
"10.211.55.13",
"k8s-master1",
"k8s-master2",
"k8s-master3",
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "BeiJing",
"L": "BeiJing",
"O": "system:kube-scheduler",
"OU": "4Paradigm"
}
]
}
Description:
The hosts list contains all kube-scheduler node IPs;
CN is system: kube-scheduler and O is system: kube-scheduler. The built-in ClusterRoleBindings system: kube-scheduler in kubernetes will give kube-scheduler the permissions required for its work.
After the above operations, we will use the following files
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json| cfssljson -bare kube-scheduler
ls | grep pem
admin-key.pem
admin.pem
ca-key.pem
ca.pem
kube-proxy-key.pem
kube-proxy.pem
kubernetes-key.pem
kubernetes.pem
kube-controller-manager-key.pem
kube-controller-manager.pem
kube-scheduler-key.pem
kube-scheduler.pem
View certificate information:
cfssl-certinfo -cert kubernetes.pem
When building a k8s cluster, distribute these files to other node machines in the cluster. At this point, the TLS certificate is created