ETCD cluster construction (available in practice)

Overview

    etcd is a key-value database with both consistency and high availability. It can be used as the backend database to save all Kubernetes cluster data.

- official website:

    Documentation versions | etcd

Prepare cfssl certificate generation tool

    cfssl is an open source certificate management tool that uses json files to generate certificates.

Operate on any server, select k8s01 here

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

Create etcd related directories

$ mkdir -pv /opt/kubernetes/etcd/{bin,cfg,ssl,data}

Create certificate

4.1 Create ca certificate json file

$ cd /opt/kubernetes/etcd/ssl

#Create ca-config

$ vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}

#Create ca-csr

$ vim ca-csr.json 
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}

4.2 Generate ca certificate

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

4.3 Use self-signed ca to issue etcd certificate

Create certificate application file:

$ cd /opt/kubernetes/etcd/ssl

# Pay attention to the hosts content. All IP addresses in the etcd cluster must be written down. You can reserve a few for future expansion.

{
    "CN": "etcd",
    "hosts": [
    "10.10.21.73",
    "10.10.21.74",
    "10.10.21.75"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

generate certificate

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# Server.pem and server-key.pem files will be generated 

$ ll |grep server
-rw-r--r-- 1 root root 1013 Sep 14 15:06 server.csr
-rw-r--r-- 1 root root  290 Sep 14 15:05 server-csr.json
-rw------- 1 root root 1679 Sep 14 15:06 server-key.pem
-rw-r--r-- 1 root root 1338 Sep 14 15:06 server.pem

Download etcd binaries

download link

https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz

Deploy ETCD cluster

6.1 Copy binary files to specified files

tar xf etcd-v3.5.0-linux-amd64.tar.gz
cp etcd-v3.5.0-linux-amd64/{etcd,etcdctl,etcdutl} /opt/kubernetes/etcd/bin

6.2 Create etcd configuration file

$vim /opt/kubernetes/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"    # k8s01为etcd-1，k8s02为etcd-2。。。每个节点唯一标识符
ETCD_DATA_DIR="/opt/kubernetes/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.241:2380,etcd-2=https://192.168.1.242:2380,etcd-3=https://192.168.1.243:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

Note:

ETCD_NAME: node name, unique in the cluster

ETCD_DATA_DIR: data directory

ETCD_LISTEN_PEER_URLS: cluster communication listening address

ETCD_LISTEN_CLIENT_URLS: Client access listening address

ETCD_INITIAL_ADVERTISE_PEERURLS: cluster advertisement address

ETCD_ADVERTISE_CLIENT_URLS: Client advertisement address

ETCD_INITIAL_CLUSTER: Cluster node address

ETCD_INITIALCLUSTER_TOKEN: Crowd Token

ETCD_INITIALCLUSTER_STATE: The current state of joining the cluster, new is a new cluster, and existing means joining an existing cluster.

6.3 Create systemd file

$ vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.targe

[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/etcd/cfg/etcd.conf
ExecStart=/opt/kubernetes/etcd/bin/etcd \
--cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--peer-key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Distribute etcd files

node02

$ scp -r /opt/kubernetes k8s02:/opt/
$ scp /etc/systemd/system/etcd.service k8s02:/etc/systemd/system/
# 记得修改etcd配置文件

node03

$ scp -r /opt/kubernetes k8s03:/opt/
$ scp /etc/systemd/system/etcd.service k8s03:/etc/systemd/system/
# 记得修改etcd配置文件

8. Start the etcd service separately

$ systemctl start etcd.service

View cluster status

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" endpoint health --write-out=table

+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://10.10.21.75:2379 |   true | 13.407895ms |       |
| https://10.10.21.74:2379 |   true |  13.61133ms |       |
| https://10.10.21.73:2379 |   true | 14.868649ms |       |
+--------------------------+--------+-------------+-------+

9. Testing

data input

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" put foo bar
OK

Read data

[root@kubenode02 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" get foo
foo
bar