Kubernetes cluster construction (binary mode)

Using the binary method to build a Kubernetes cluster allows you to customize and configure Kubernetes more flexibly and freely. At the same time, it can also achieve higher performance and smaller resource footprint.

For a beginner like me:

• See the various components of Kubernetes more intuitively and understand the relationships and functions between them.
• In the process of building a Kubernetes cluster, understanding the cluster's architecture and the collaborative relationship between various components will help beginners better understand the working principles and mechanisms of Kubernetes.
• Building a Kubernetes cluster using the binary method requires some configuration and maintenance work, which helps beginners master Kubernetes maintenance skills and improve their technical level.

1. Installation requirements

Before starting, the following conditions need to be met to deploy Kubernetes cluster machines:

(1) One or more machines, operating system CentOS7.x-86_x64

(2) Hardware configuration: 2GB or more RAM, 2 CPUs or more CPUs, hard disk 30GB or more

(3) Network communication between all machines in the cluster

(4) If you can access the external network, you need to pull the image. If the server cannot access the Internet, you need to download the image in advance and import it into the node.

(5) Disable swap partition

2. Prepare the environment

(1) Software environment:

operating system	CentOS7.8_x64
Docker	19-this
Kubernetes	1.19

(2) Server planning

Role	IP	components
k8s-master	192.168.122.143	for apiserver, for controller-manager, for scheduler, etcd
k8s-node1	192.168.122.144	kubelet， kube-proxy， docker etcd
k8s-node2	192.168.122.145	kubelet， kube-proxy， docker， etcd

3. Operating system initialization configuration

Three linux systems are operated by default

1. Turn off the firewall:

Firewalls filter and block network data packets, which may affect communication between Kubernetes cluster nodes. At the same time, Kubernetes itself has a relatively complete network policy mechanism to ensure the network security of the cluster, so turning off the firewall will not affect the security of the Kubernetes cluster.

systemctl stop firewalld

systemctl disable firewalld

2. Turn off selinux:

In a Kubernetes cluster, SELinux is an optional security module that provides mandatory access control and access auditing capabilities. However, when building a Kubernetes cluster, in order to simplify configuration and avoid possible problems, many administrators choose to turn off SELinux. This is mainly because:

1. SELinux has strict access control for containers, which may cause some applications to fail to work properly or be unable to access necessary resources.
2. In some cases, the rules of SELinux do not fit well with the Kubernetes cluster installation configuration, which can cause problems and errors.
3. Turning off SELinux can simplify configuration and management work, making cluster deployment and maintenance more convenient. However, turning off SELinux will also reduce the security and reliability of the cluster. SELinux must be re-enabled when necessary.

Therefore, turning off SELinux can make the deployment of a Kubernetes cluster simpler and more reliable, but it can also reduce the security and reliability of the cluster. In actual applications, it is necessary to determine whether SELinux needs to be turned on or off based on specific circumstances.

sed -i ‘s/enforcing/disabled/’ /etc/selinux/config # 永久

setenforce 0 # temporary

3. Close swap:

Kubernetes uses cgroups to manage container resources. A swap partition may prevent the container from using expected memory resources and may cause the application to crash or have other issues inside the container.

Kubernetes itself does not use swap. At the same time, because the use of containers and the mechanism of swapping memory are different, if an application needs to use a large amount of memory, the container will automatically apply for more memory instead of using swap, avoiding performance loss and inability to Predicted behavior. Turning off the swap partition can better protect the stability and performance of the Kubernetes cluster and ensure consistent memory usage and performance of the container.

swapoff -a # temporary

sed -ri ‘s/.swap./#&/’ /etc/fstab # permanent

4. Set the host name according to the plan:

hostnamectl set-hostname

192.168.122.143 ：

192.168.122.144：

192.168.122.145：

5. Add hosts to the master node:

cat >> /etc/hosts << EOF
192.168.122.143 master
192.168.122.144 node1
192.168.122.145 node2
EOF

6 Pass bridged IPv4 traffic to the iptables chain:

In a Kubernetes cluster, each Pod is assigned an IP address, and the containers within the Pod are also assigned a virtual network card and IP address. When two Pods need to communicate with each other, they communicate using these IP addresses.

However, when containers within a Pod try to communicate with another Pod, they do not send packets directly using its IP address, but instead use a bridge to communicate. This means that packets will be transmitted through the bridge device in the Linux kernel instead of being sent through the network interface.

In order to ensure that these bridged packets can be routed and forwarded correctly, they need to be passed to the iptables chain for processing. Iptables can be used to define network rules so that packets can be routed correctly to their destination. By passing bridged IPv4 traffic into the iptables chain, you can ensure that Pods in the Kubernetes cluster can communicate correctly, and you can implement some advanced network functions, such as network policy and load balancing.

Each node must execute

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

Make the configuration effective

sysctl --system # take effect

7. Time synchronization:

Each node in the Kubernetes cluster needs to communicate with each other and collaborate, so their time needs to be synchronized to ensure that they can accurately coordinate their work when planning and scheduling. If the nodes' times are out of sync, the following issues may occur:

1. Unpredictable errors occur in container operation.
2. The scheduler cannot accurately calculate the completion time of tasks, causing tasks to time out or be scheduled on inappropriate nodes.
3. Monitoring and log collection systems may be time misaligned, resulting in inaccurate data analysis results.

Therefore, in order to ensure the normal operation of the cluster, the time needs to be synchronized on each node in the cluster.

yum install ntpdate -y

ntpdate time.windows.com

Note: After configuring the above commands, it is best to restart Linux to ensure that the configuration takes effect.

4. Deploy the etcd cluster

Etcd is a distributed key-value storage system. Kubernetes uses Etcd for data storage, so first prepare an Etcd database. In order to solve the single point of failure of Etcd, it should be deployed in a cluster. Here, 3 machines are used to form a cluster, which can tolerate the failure of 1 machine. , Of course, you can also use 5 machines to form a cluster, which can tolerate the failure of 2 machines.

Node name	IP
etcd-1	192.168.122.143
etcd-2	192.168.122.144
etcd-3	192.168.122.145

Note: In order to save machines, this is reused with K8s node machines. It can also be deployed independently of the k8s cluster, as long as the apiserver can be connected.

4.1 Prepare the cfssl certificate generation tool

CFSSL is a Go-based certificate and key management tool that can be used to quickly and easily generate, sign and verify TLS certificates. CFSSL supports a variety of certificate formats and encryption algorithms, and provides an extensible API that allows it to be easily integrated with other tools and applications.

You can find any server to operate. The Master node is used here.

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

4.2 Generate etcd certificate

(1) Self-signed Certificate Authority (CA)

Create working directory:

mkdir -p ~/TLS/{etcd,k8s}

cd TLS/etcd

Self-signed CA:

The signing configuration, including default expiration times and different certificate profiles, is defined in the ca-config.json file. Each certificate profile has its own set of uses and expiration times.

cat > ca-config.json<< EOF
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "www": {
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

ca-csr.json is a JSON file containing CA certificate request information, and ca is the generated file prefix.

The certificate request of etcd CA is defined in the ca-csr.json file, including name and key.

cat > ca-csr.json<< EOF
{
    "CN": "etcd CA",
    "key": {
    "algo": "rsa",
    "size": 2048
    },
    "names": [{
    "C": "CN",
    "L": "Beijing",
    "ST": "Beijing"
    }]
}
EOF

Generation certificate:

Use the CFSSL command line tool to generate CA certificates and keys.

cfssl gencert -initca ca-csr.json | cfssljson -bare ca - ls *pem

ca.pem and ca-key.pem are CA certificate and key files

(2) Use self-signed CA to issue Etcd HTTPS certificate

Create certificate application file:

server-csr.json is a JSON file containing server certificate request information

cat > server-csr.json<< EOF
{
    "CN": "etcd",
    "hosts": [
        "192.168.122.143",
        "192.168.122.144",
        "192.168.122.145"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
        "C": "CN",
        "L": "BeiJing",
        "ST": "BeiJing"
    }]
}
EOF

Note: The IP in the hosts field of the above file is the cluster internal communication IP of all etcd nodes. No one is missing! In order to facilitate later expansion, you can write a few more reserved IPs.

(3) Generate certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

Check

ls server*pem

server.pem is the server's certificate file, also called a public key certificate, used to verify the server's identity; server-key.pem is the server's private key file, used to decrypt the encrypted data sent by the client.

4.3 Download binaries from Github

Download address: https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

4.4 Deploy Etcd cluster

The following operation is performed on node 1. To simplify the operation, all files generated by node 1 will be copied to node 2 and node 3 later.

(1) Unzip the binary package

tar zxvf etcd-v3.4.9-linux-amd64.tar.gz

Create working directory

mkdir -p /opt/etcd/{bin,cfg,ssl}

mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

(2) Create etcd configuration file

Various parameters of etcd can be configured through the etcd.conf file, including the address of the cluster node, data storage path, listening port, certificate configuration, etc.

What is configured here is the configuration of master:192.168.122.143 etcd

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.122.143:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.122.143:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.122.143:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.122.143:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.122.143:2380,etcd-2=https://192.168.122.144:2380,etcd-3=https://192.168.122.145:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME: Node name, unique in the cluster

ETCD_DATA_DIR: Data directory

ETCD_LISTEN_PEER_URLS: Cluster communication listening address

ETCD_LISTEN_CLIENT_URLS: Client access listening address

ETCD_INITIAL_ADVERTISE_PEER_URLS: Cluster advertisement address

ETCD_ADVERTISE_CLIENT_URLS: Client advertisement address

ETCD_INITIAL_CLUSTER: Cluster node address

ETCD_INITIAL_CLUSTER_TOKEN: Cluster Token

ETCD_INITIAL_CLUSTER_STATE: The current status of joining the cluster, new is a new cluster, and existing means joining an existing cluster.

(3) systemd manages etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
    --cert-file=/opt/etcd/ssl/server.pem \
    --key-file=/opt/etcd/ssl/server-key.pem \
    --peer-cert-file=/opt/etcd/ssl/server.pem \
    --peer-key-file=/opt/etcd/ssl/server-key.pem \
    --trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

(4) Copy the certificate just generated

Copy the certificate just generated to the path in the configuration file

cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

Check whether it is copied to the specified path

(5) Copy all files generated by node 1 above to node 2 and node 3

Copy the master-related etcd files to ndoe1 and node2

scp -r /opt/etcd/ [email protected]:/opt/etcd/

scp -r /opt/etcd/ [email protected]:/opt/etcd/

scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/

scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/

Here you must confirm whether the file has been copied, please go to the specified directory to check

Then modify the node name and current server IP in the etcd.conf configuration file on node 2 and node 3 respectively:

node2:192.168.122.144

vi /opt/etcd/cfg/etcd.conf

content

#[Member]
ETCD_NAME="etcd-2" # 修改此处， 节点 2 改为 etcd-2， 节点 3 改为 etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.122.144:2380" # 修改此处为当前服务器 IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.122.144:2379" # 修改此处为当前服务器 IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.122.144:2380" # 修改此处为当前服务器 IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.122.144:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.122.143:2380,etcd-2=https://192.168.122.144:2380,etcd-3=https://192.168.122.145:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node3:192.168.122.145

vi /opt/etcd/cfg/etcd.conf

content

#[Member]
ETCD_NAME="etcd-3" # 修改此处， 节点 2 改为 etcd-2， 节点 3 改为 etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.122.145:2380" # 修改此处为当前服务器 IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.122.145:2379" # 修改此处为当前服务器 IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.122.145:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.122.145:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.122.143:2380,etcd-2=https://192.168.122.144:2380,etcd-3=https://192.168.122.145:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

Be sure to determine whether there are errors such as wrong line breaks here. It is best to remove the comments to prevent interference.

(6) Start and set startup

Master, node1, and node2 execute systemctl start etcd in sequence. The master will wait for node1 or node2 to start.

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd

The systemctl daemon-reload command is used to reload the systemd daemon configuration file. When we modify the systemd configuration file, we need to use this command to make the new configuration take effect.

The systemctl start etcd command is used to start the etcd service.

The systemctl enable etcd command is used to set the etcd service to start automatically at boot.

mistake

Error when starting etcd from node

Please confirm whether steps 5 and 6 are correct. What I have here is caused by incorrect configuration of /opt/etcd/cfg/etcd.conf

(7) View cluster status

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints=192.168.122.143:2379,192.168.122.144:2379,192.168.122.145:2379 endpoint health

• ETCDCTL_API=3: Set the API version of etcdctl tool to 3, which is the API used by etcd 3.x version.
• /opt/etcd/bin/etcdctl: The path to execute etcdctl tool.
• –cacert=/opt/etcd/ssl/ca.pem: Specify the root certificate path of etcd cluster.
• –cert=/opt/etcd/ssl/server.pem: Specify the server certificate path of etcd cluster.
• –key=/opt/etcd/ssl/server-key.pem: Specify the private key path of the etcd cluster.
• –endpoints=192.168.122.143:2379,192.168.122.144:2379,192.168.122.145:2379: Specify the node address of etcd cluster. Multiple nodes are separated by commas.
• endpoint health: command executed to check whether each node of the etcd cluster is healthy. If unhealthy is returned, there is a problem with the node.

If the above information is output, it means that the cluster deployment is successful. If there is a problem, the first step is to read the log: /var/log/message or journalctl -u etcd

Generally, it is caused by /opt/etcd/cfg/etcd.conf configuration error or file copy error.

5. Deploy Master Node

5.1 Generate kube-apiserver certificate

(1) Self-signed Certificate Authority (CA)

cat > kube-proxy-csr.json<< EOF
{
    
    
    "CN": "system:kube-proxy",
        "hosts":{
    
    },
    "key": {
    
    
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
    
    
        "C": "CN",
        "L": "Beijing",
        "ST": "Beijing",
        "O": "k8s",
        "OU": "System"
    }]
}
EOF

cat > ca-csr.json<< EOF
{
    
    
    "CN": "kubernetes",
    "key": {
    
    
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
    
    
        "C": "CN",
        "L": "Beijing",
        "ST": "Beijing",
        "O": "k8s",
        "OU": "System"
    }]
}
EOF

cat > ca-config.json<< EOF
{
    
    
    "signing": {
    
    
        "default": {
    
    
            "expiry": "87600h"
        },
        "profiles": {
    
    
            "kubernetes": {
    
    
                "expiry": "87600h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}
EOF

(2) Generate certificate:

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

Check

ls *pem

(3) Use a self-signed CA to issue the kube-apiserver HTTPS certificate

Create certificate application file:

cd TLS/k8s
cat > server-csr.json<< EOF
{
    
    
    "CN": "kubernetes",
    "hosts": [
        "10.0.0.1",
        "127.0.0.1",
        "192.168.122.143",
        "192.168.122.144",
        "192.168.122.145",
              "192.168.122.146",
        "192.168.122.147",
        "192.168.122.148",

        "kubernetes",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local"
    ],
    "key": {
    
    
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
    
    
        "C": "CN",
        "L": "BeiJing",
        "ST": "BeiJing",
        "O": "k8s",
        "OU": "System"
    }]
}
EOF

Generate certificate:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json |cfssljson -bare server

5.2 Download binaries from Github

Download address: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md#v1183

Note: When you open the link, you will find that there are many packages inside. It is enough to download a server package, which contains the Master and Worker Node binary files.

Upload to server

5.3 Unzip the binary package

tar zxvf kubernetes-server-linux-amd64.tar.gz

mkdir -p /opt/kubernetes/{
    
    bin,cfg,ssl,logs}

cd kubernetes/server/bin

Copy files

cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
cp kubectl /usr/bin/

5.4 Deploy kube-apiserver

1. Building arrangement text item

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.122.143:2379,https://192.168.122.144:2379,https://192.168.122.145:2379 \\
--bind-address=192.168.122.143 \\
--secure-port=6443 \\
--advertise-address=192.168.122.143 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

Note: The first of the above two \ \ is an escape character, and the second one is a newline character. The escape character is used to preserve the newline character using EOF.

–logtostderr: enable logging

-v: log level

–log-dir: log directory

–etcd-servers: etcd cluster address

–bind-address: listening address

–secure-port: https secure port

–advertise-address: cluster advertisement address

–allow-privileged: enable authorization

–service-cluster-ip-range: Service virtual IP address range

–enable-admission-plugins: admission control module

–authorization-mode: Authentication and authorization, enable RBAC authorization and node self-management

–enable-bootstrap-token-auth: Enable TLS bootstrap mechanism

–token-auth-file: bootstrap token text

–service-node-port-range: Service nodeport type default assigned port range

–kubelet-client-xxx: apiserver access kubelet client certificate

–tls-xxx-file: apiserver https certificate

–etcd-xxxfile: Connect Etcd cluster certificate

–audit-log-xxx: audit log

2. Copy the certificate just generated

Copy the certificate just generated to the path in the configuration file:

cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

3. 启用 TLS Bootstrappin

TLS Bootstraping: After Master apiserver enables TLS authentication, the Node node kubelet and kube-proxy must use a valid certificate issued by the CA to communicate with the kube-apiserver. When there are many Node nodes, this kind of client certificate issuance requires a lot of work. , which will also increase the complexity of cluster expansion. In order to simplify the process, Kubernetes introduces the TLS bootstraping mechanism to automatically issue client certificates. The kubelet will automatically apply for a certificate from the apiserver as a low-privilege user. The kubelet's certificate is dynamically signed by the apiserver.

Therefore, it is strongly recommended to use this method on Node. Currently, it is mainly used for kubelet. We still issue a unified certificate for kube-proxy.

TLS bootstrapping workflow:

Create the token file in the above configuration file:

cat > /opt/kubernetes/cfg/token.csv << EOF
c47ffb939f5ca36231d9e3121a252940,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

Format: token, username, UID, user group

4. systemd management apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

5. Start and set up startup

systemctl daemon-reload
systemctl start kube-apiserver
systemctl enable kube-apiserver

6. Authorize the kubelet-bootstrap user to allow requesting certificates

kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

5.5 Deploy kube-controller-manager

1. Create configuration file

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--master=127.0.0.1:8080 \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"
EOF

– master: Connect to the apiserver via the local non-secure local port 8080.

– leader-elect: Automatic election (HA) when this component starts multiple

– cluster-signing-cert-file/– cluster-signing-key-file: CA that automatically issues certificates for kubelet, consistent with apiserver

2. systemd management controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3. Start and set up startup

systemctl daemon-reload
systemctl start kube-controller-manager
systemctl enable kube-controller-manager

5.6 Deploy kube-scheduler

1. Create configuration file

cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF

– master: Connect to the apiserver via the local non-secure local port 8080.

– leader-elect: Automatic election (HA) when this component starts multiple

2. systemd management scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3. Start and set up startup

systemctl daemon-reload
systemctl start kube-scheduler
systemctl enable kube-scheduler

4. Check cluster status

All components have been started successfully. Check the current cluster component status through the kubectl tool:

kubectl get cs

See why

journalctl -xe

I found that controller-manage failed to start and checked the reason. It turned out that there was a missing space.

Restart after modification

View again

The above output shows that the Master node component is running normally.

6. Install Docker

Download address: https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz

The following operates on all nodes. Binary installation is used here, and installation with yum is the same. Note: Since the master needs to install kubelet later, the master also needs to install docker.

Only the commands for the node1 node are shown here.

There must be some friends who want to ask why the master does not install docker?

Because the roles of Master node and Worker node are different.

When building a Kubernetes cluster in binary mode, components such as kube-apiserver, kube-scheduler, kube-controller-manager, etcd are usually installed on the Master node instead of installing Docker. This is because the Master node does not directly run containers. Its main responsibility is to manage and schedule containers, so it only needs to rely on Docker or other container runtimes.

(1) Unzip the binary package

Unzip

tar zxvf docker-19.03.9.tgz

Move all files in the docker directory to the /usr/bin directory.

mv docker/* /usr/bin

(2) systemd manages docker

docker.service is the service unit file of the Docker service. Its function is to define the startup method, operating environment and related dependencies of the Docker service, so that the operating system can correctly start the Docker service at startup.

The systemd service manager automatically loads these service unit files and manages system services according to the rules defined in them.

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

(3) Create configuration file

The configuration file of the Docker daemon, which is used to configure the behavior and properties of the Docker daemon.

The Docker image warehouse address is configured here.

mkdir /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"registry-mirrors": ["https://vpmkvcwz.mirror.aliyuncs.com"]
} 
EOF

registry-mirrors Alibaba Cloud mirror site

(4) Start and set startup

systemctl daemon-reload
systemctl start docker
systemctl enable docker

View docker information

docker info

[root@node1 ~]# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.9
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-1160.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 972.3MiB
 Name: node1
 ID: DUDF:D4US:NWFZ:SG2O:2QUY:UWNX:DYUH:IEG5:CG5E:F6ND:JGVK:BX34
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://vpmkvcwz.mirror.aliyuncs.com/
 Live Restore Enabled: false
 Product License: Community Engine

7. Deploy Worker Node

The master node is also deployed as a worker node and participates in scheduling.

And perform unified configuration on the master, and then copy the configuration to the worke node for rapid deployment

7.1 Create a working directory and copy binary files

Create working directories on all worker nodes:

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

Copy from the master node:

cd kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes/bin

7.2 Deploy kubelet

kubelet: The kubelet on each Node node regularly calls the REST interface of the API Server to report its own status. After receiving this information, the API Server updates the node status information to etcd. Kubelet also monitors Pod information through the API Server to manage Pods on the Node machine, such as creating, deleting, and updating Pods.

Even if you do not need to run the container directly on the master node, you can still deploy kubelet to ensure the normal operation of the Kubernetes cluster.

1. Building arrangement text item

cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=master \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet-config.yml \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=lizhenliang/pause-amd64:3.0"
EOF

–hostname-override: display name, unique in the cluster

–network-plugin: enable CNI

–kubeconfig: empty path, which will be automatically generated and later used to connect to apiserver

–bootstrap-kubeconfig: Apply for a certificate from apiserver when starting for the first time

–config: configuration parameter file

–cert-dir: kubelet certificate generation directory

–pod-infra-container-image: manages the image of the Pod network container

2. Arrangement reference sentence item

Please note the empty lines in the yml format

cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
  - 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s
evictionHard:
  imagefs.available: 15%
  memory.available: 100Mi
  nodefs.available: 10%
  nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110
EOF

3. Generate bootstrap.kubeconfig Text item

This file is used to create a kubelet-bootstrap user to start the kubelet process and register it into the cluster

Specify the address and port of the Kubernetes API server.

KUBE_APISERVER="https://192.168.122.143:6443" # apiserver IP:PORT

The token is consistent with token.csv

TOKEN="c47ffb939f5ca36231d9e3121a252940"

View token value

cat /opt/kubernetes/cfg/token.csv

Generate kubelet bootstrap kubeconfig configuration file

kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

kubectl config set-credentials "kubelet-bootstrap" \
--token=${TOKEN} \
--kubeconfig=bootstrap.kubeconfig

kubectl config set-context default \
--cluster=kubernetes \
--user="kubelet-bootstrap" \
--kubeconfig=bootstrap.kubeconfig

kubectl config use-context default \
--kubeconfig=bootstrap.kubeconfig

This command can be executed together and can be connected through; or &&

Copy to configuration file path:

cp bootstrap.kubeconfig /opt/kubernetes/cfg

4. systemd management kubelet

cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5. Start and set up startup

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet

Report an error

1. Note: Kubernetes officially recommends that docker and others use systemd as the Cgroup Driver. If the kubelet service status is abnormal and the log shows that the cgroupDriver does not match, first check the Cgroup Driver used by docker.

docker info|grep “Cgroup Driver”

Then modify the parameters of the Cgroup Driver in the yaml file of kubelet

2.启动失败failed to run Kubelet: failed to get docker version: Cannot connect to the Docker daemon at unix:///var/run/docker.s

At first I thought that docker failed to start or there was a problem with the deamon.json configuration. After confirmation, no problem was found. Finally, I checkedsystemd Management a> kubelet'skubelet.service file found one more\

After modification

Restart

7.3 Approve kubelet certificate application and join the cluster

Each node needs to be approved after deployment application before it can join the cluster.

# View kubelet certificate request

kubectl get csr

#Approval application

kubectl certificate approve node-csr-VXXOMedFY9mQu9_wh2t64wqeBXHRJL8H68DfougGySA

# View nodes

kubectl get node

Note: Since the network plug-in has not been deployed yet, the node will not be ready NotReady

7.4 Deploy kube-proxy

1. Building arrangement text item

cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"
EOF

2. Arrangement reference sentence item

cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: master
clusterCIDR: 10.0.0.0/24
EOF

3. Generate kube-proxy.kubeconfig Text item

Create kube-proxy Verification:

# Switch working directory

cd /root/TLS/k8s

#Create certificate request file

cat > kube-proxy-csr.json<< EOF
{
    
    
    "CN": "system:kube-proxy",
    "hosts": [],
    "key": {
    
    
        "algo": "rsa",
        "size": 2048
    },
    "names": [{
    
    
        "C": "CN",
        "L": "BeiJing",
        "ST": "BeiJing",
        "O": "k8s",
        "OU": "System"
    }]
}
EOF

# Generate certificate

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -
profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

Check

ls kube-proxy*pem

Create kubeconfig Text:

KUBE_APISERVER="https://192.168.122.143:6443"

kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=${KUBE_APISERVER} --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy --client-certificate=./kube-proxy.pem --client-key=./kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

Copy to the specified path in the configuration file:

cp kube-proxy.kubeconfig /opt/kubernetes/cfg/

4. systemd 管理 kube-proxy

cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

5. Start and set up startup

systemctl daemon-reload
systemctl start kube-proxy
systemctl enable kube-proxy

Report an error

1. Prompt invalid configuration: no server found for cluster “kubernetes”

This is because this command was not executed when generating the kubeconfig file.

KUBE_APISERVER="https://192.168.122.143:6443"

After execution, regenerate the kubeconfig file and restart kube-proxy.

7.5 Deploy CNI network

CNI is the abbreviation of Container Network Interface. It is a specification that provides a plug-in network configuration and management mechanism for container runtimes (such as Docker, Kubernetes, etc.). The CNI plug-in is responsible for implementing network configuration, management and connection, including IP address allocation, routing settings, network isolation and other functions. Through the CNI plug-in, containers can be connected to different types of networks, such as physical networks, virtual networks, overlay networks, etc. In Kubernetes, CNI is used as the management interface of the container network and can implement a variety of different network solutions, such as Flannel, Calico, Cilium, etc.

First prepare the CNI binary file:

download link:

https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz

Unzip the binary package and move to the default working directory:

mkdir /opt/cni/bin

tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

Deploy the CNI network:

wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

If the default image address is inaccessible, change it to the docker hub image warehouse.

sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

start up

kubectl apply -f kube-flannel.yml

kubectl get node

Deploy the network plug-in and Node is ready

kubectl get pods -n kube-system

The pod cannot be obtained here because the node deployment has not started yet, so skip it first.

7.6 Authorize apiserver to access kubelet

cat > apiserver-to-kubelet-rbac.yaml<< EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
  - ""
  resources:
  - nodes/proxy
  - nodes/stats
  - nodes/log
  - nodes/spec
  - nodes/metrics
  - pods/log
  verbs:
  - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: kubernetes
EOF

Authorize access to kubelet

kubectl apply -f apiserver-to-kubelet-rbac.yaml

Check

kubectl get ClusterRole

7.7 Newly added Worker Node

Here is the actual deploymentWorker Node operation

1. Copy the deployed Node related files to the new node in mster

Copy the Worker Node related files on the master node to nodes 192.168.122.144 and 192.168.122.145

Only the operations of the node1 node are shown here.

scp -r /opt/kubernetes [email protected]:/opt/

scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service [email protected]:/usr/lib/systemd/system

scp -r /opt/cni/ [email protected]:/opt/

scp /opt/kubernetes/ssl/ca.pem [email protected]:/opt/kubernetes/ssl

2. kubeletconfirmation kubeconfigtext

rm /opt/kubernetes/cfg/kubelet.kubeconfig
rm -f /opt/kubernetes/ssl/kubelet*

Note: These files are automatically generated after the certificate application is approved. Each Node is different and must be deleted and regenerated. Otherwise, the master node cannot obtain the node node information.

3. Repair main machine name

vi /opt/kubernetes/cfg/kubelet.conf

Modify hostname

--hostname-override=node1

vi /opt/kubernetes/cfg/kube-proxy-config.yml

Modify hostname

hostnameOverride: node1

4. Start and set up startup

systemctl daemon-reload
systemctl start kubelet
systemctl enable kubelet
systemctl start kube-proxy
systemctl enable kube-proxy

Report an error

1. Check the status and find that kubectl failed to start.

To view specific issues, use the journalctl -xe command

I thought it might be that the node did not start docker.

Restart, ok

Pay attention hereSet the image accelerator address

5. current Master approved new Node kubelet Confirmation Book

View certificate request

kubectl get csr

Authorization request

kubectl certificate approve node-csr-TI8Mrdx61UpWpuPJswDzUngNjivBM5rMw4qZbfTT6PA

6. View Node Status

kubectl get node

Node2 (192.168.122.145) node is the same as above. Remember to change the hostname!

Continue to deploy node2 to cooperate, the operation is the same as above

8. Test the kubernetes cluster

Create a pod in the Kubernetes cluster and verify whether it is running normally [master node operation]

8.1 Download nginx [Be able to connect to the Internet to pull the nginx image]

kubectl create deployment nginx --image=nginx

View status

kubectl get pod

If we appear in the Running state, it means that it has been successfully run.

8.2 Next we need to expose the port so that other outsiders can access it.

exposed port

kubectl expose deployment nginx --port=80 --type=NodePort

Check the external port

kubectl get pods,svc

As you can see, we have successfully exposed port 80 to 31167

We go to our host browser and visit the following address 192.168.122.143:31167

Found that our nginx has been successfully started

delete

kubectl delete deployment nginx

Delete the deployment named "nginx" and the associated pods and replica sets from the Kubernetes cluster.