一步一步搞定Kubernetes二进制部署（一）——etcd集群搭建（单节点）

前言

前面简单介绍了Kubernetes的基础理论原理与核心组件，本文将给出单节点以二进制部署Kubernetes集群的第一个步骤——搭建etcd集群实验流程。

集群规划清单（之后的文章将按照该规划一步一步配置）

1、三个节点上部署etcd集群
2、两个node节点上部署docker环境、部署flannel(容器内部之间的通信，依赖于vxlan技术)
3、master节点上部署kube-apiserver、kube-controller-manager、kube-scheduler
4、node节点上部署kubelet、kube-proxy

服务器ip地址规划

master01地址：192.168.0.128

node01地址：192.168.0.129

node02地址：192.168.0.130

本文将给出的是etcd集群的搭建配置流程。

搭建流程

1、环境准备

三台服务器建议设置主机名称、绑定静态ip和关闭网络管理服务，其次需要关闭防火墙和核心防护，清空iptables

这里以master01为例，其他节点按照此节点类似设置即可

[root@localhost ~]# hostnamectl set-hostname master01
[root@localhost ~]# su
[root@master01 ~]# systemctl stop firewalld
[root@master01 ~]# setenforce 0
[root@master01 ~]# iptables -F

设置静态ip地址

[root@master01 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33 
[root@master01 ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="7c933cbb-b29c-4a36-bb12-d1ac1c505524"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="192.168.0.128"
NETMASK="255.255.255.0"
GATEWAY="192.168.0.2"
DNS1=192.168.0.2
[root@master01 ~]# systemctl restart network
[root@master01 ~]# ip add
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:06:97:04 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.128/24 brd 192.168.0.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::5bd9:44c7:cf2c:ef20/64 scope link 
       valid_lft forever preferred_lft forever
[root@master01 ~]# ping www.baidu.com #这里测试是否可以连外网
PING www.a.shifen.com (180.101.49.12) 56(84) bytes of data.
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=1 ttl=128 time=13.4 ms
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=2 ttl=128 time=11.2 ms
64 bytes from 180.101.49.12 (180.101.49.12): icmp_seq=3 ttl=128 time=11.1 ms
^C
--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2008ms
rtt min/avg/max/mdev = 11.135/11.967/13.476/1.068 ms
#关闭网络管理
[root@master01 ~]# systemctl stop NetworkManager
[root@master01 ~]# systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.

2、通过下载相关命令工具来搞定ca证书的创建、生成、查看

首先、在master节点上,安装制作查看证书的相关工具，如：cfssl、cfssljson、cfssl-certinfo

//先自己创建并且进入工作目录
[root@master01 ~]# mkdir k8s
[root@master01 ~]# cd k8s/
#编写脚本下载上述三个命令工具到指定的目录下
[root@master01 k8s]# vim cfssl.sh #脚本在下一个代码块中另外给出
[root@master01 k8s]# bash cfssl.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9.8M  100  9.8M    0     0   131k      0  0:01:17  0:01:17 --:--:-- 47667
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2224k  100 2224k    0     0   823k      0  0:00:02  0:00:02 --:--:--  823k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6440k  100 6440k    0     0   426k      0  0:00:15  0:00:15 --:--:--  523k

下载命令的shell脚本如下：

[root@master01 k8s]# cat cfssl.sh 
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson 
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo 
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo
#查看命令是否成功安装到指定目录
[root@master01 k8s]# ls /usr/local/bin/
cfssl  cfssl-certinfo  cfssljson

用于创建生成ca证书的工具已经有啦，现在就需要材料（各种文件）来创建证书了

创建证书生成目录，从而生成各种需要的证书

[root@master01 k8s]# mkdir etcd-cert
[root@master01 k8s]# cd etcd-cert/
[root@master01 etcd-cert]# ls
#编写证书创建需要的材料——各种文件，可以依赖shell脚本一键执行，下一个代码块将解释脚本的主要内容及相关参数
[root@master01 etcd-cert]# vim etcd-cert.sh
#执行脚本
[root@master01 etcd-cert]# bash etcd-cert.sh 
2020/05/03 20:05:57 [INFO] generating a new CA key and certificate from CSR
2020/05/03 20:05:57 [INFO] generate received request
2020/05/03 20:05:57 [INFO] received CSR
2020/05/03 20:05:57 [INFO] generating key: rsa-2048
2020/05/03 20:05:57 [INFO] encoded CSR
2020/05/03 20:05:57 [INFO] signed certificate with serial number 594719677485784071979843988457153533430072455164
2020/05/03 20:05:57 [INFO] generate received request
2020/05/03 20:05:57 [INFO] received CSR
2020/05/03 20:05:57 [INFO] generating key: rsa-2048
2020/05/03 20:05:58 [INFO] encoded CSR
2020/05/03 20:05:58 [INFO] signed certificate with serial number 603393813825663113730743133623713339445920555574
2020/05/03 20:05:58 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

#查看通过脚本执行产生的文件，下面将剖析这些文件的由来和相关内容解释
[root@master01 etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem

shell脚本文件和相关解释（复制脚本后先取消我的注释，否则会报错）

#编写ca证书配置文件，第一个json文件
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h" #过期时间为10年
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth", #服务和客户端的授权验证 表示client可以用该 CA 对server提供的证书进行验证；
            "client auth" #表示server可以用该CA对client提供的证书进行验证；
        ]
      }
    }
  }
}
EOF
#编写ca证书签名文件，第二个json文件
cat > ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",#非对称密钥格式
        "size": 2048 #字节长度
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF
#使用下载的命令依赖编写的json文件来生成ca证书，从而生成两个pem证书文件：ca.pem和ca-key.pem
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#-----------------------
#指定etcd集群中三个节点之间的通信验证，需要编写server端的验证签名文件第三个json文件
cat > server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.0.128",
    "192.168.0.129",
    "192.168.0.130"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}
EOF
#使用cfssl命令依赖ca证书和配置文件，-profiles是指定特定的使用场景，将会生成：
# server-key.pem 和 server.pem证书（私钥）
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

梳理一下，脚本为一个.sh文件,脚本中编写了三个json文件，通过命令生成4个证书（私钥）文件（一对为ca的、另一对为server端的）以及对应的签名文件（ca.csr和server.csr文件）

以上我们需要的证书就创建完毕了，下面开始etcd集群的搭建了

3、根据证书以及软件包和脚本搭建etcd集群

首先我们需要下载etcd软件包，这边我使用第三方工具从官方上下载下来的，可以给大家使用，软件包链接

链接：https://pan.baidu.com/s/1jMeYk2oAYNE9woDBlgnZYA
提取码：wwz4

etcd软件包的网站链接也贴出来：ETCD 二进制包地址

https://github.com/etcd-io/etcd/releases

资源给出来了下面开始干活吧！^_^

首先解压该软件包

[root@master01 k8s]# ls
cfssl.sh  etcd-cert  etcd-v3.3.10-linux-amd64.tar.gz
[root@master01 k8s]# tar zxf etcd-v3.3.10-linux-amd64.tar.gz 
[root@master01 k8s]# cd etcd-v3.3.10-linux-amd64/
#查看一下软件包中的文件
[root@master01 etcd-v3.3.10-linux-amd64]# ls
Documentation  etcd  etcdctl  README-etcdctl.md  README.md  READMEv2-etcdctl.md

#我们需要的是etcd和etcdctl两个命令工具待会我们需要将它们移动到我们创建的集群搭建目录中，现在我们创建各种命令，然后将需要的文件存入或创建写入对应的目录中

首先创建集群搭建的各种目录

[root@master01 ~]# mkdir /opt/etcd/{cfg,bin,ssl} -p 
#分别为配置文件目录，命令目录和证书目录
[root@master01 ~]# ls -R /opt/etcd/
/opt/etcd/:
bin  cfg  ssl

/opt/etcd/bin:

/opt/etcd/cfg:

/opt/etcd/ssl:
#目前目录中没有任何文件

移动相关文件

[root@master01 etcd-v3.3.10-linux-amd64]# mv etcd etcdctl /opt/etcd/bin/
[root@master01 etcd-v3.3.10-linux-amd64]# cd /root/k8s/etcd-cert/
[root@master01 etcd-cert]# ls
ca-config.json  ca-csr.json  ca.pem        server.csr       server-key.pem
ca.csr          ca-key.pem   etcd-cert.sh  server-csr.json  server.pem
[root@master01 etcd-cert]# cp *.pem /opt/etcd/ssl
[root@master01 etcd-cert]# ls -R /opt/etcd/
/opt/etcd/:
bin  cfg  ssl

/opt/etcd/bin:
etcd  etcdctl

/opt/etcd/cfg:

/opt/etcd/ssl:
ca-key.pem  ca.pem  server-key.pem  server.pem

现在就差配置文件（包括配置文件和服务启动文件）了，这里依旧可以使用shell脚本（真正体会到shell脚本的魅力了吧）来编写，务必注意将其中的注释去除后再执行该脚本

[root@master01 etcd-cert]# cd /opt/etcd/cfg/
[root@master01 cfg]# ls
[root@master01 cfg]# vim etcd.sh
#!/bin/bash
# example: ./etcd.sh etcd01 192.168.1.10 etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380

#位置变量
ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3
#工作路径
WORK_DIR=/opt/etcd
#重定向写入文件
cat <<EOF >$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380" #etcd群集服务器之间通信端口
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"#外部访问节点的端口

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #令牌名称集群节点服务器需要保持一致
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

cat <<EOF >/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--name=\${ETCD_NAME} \
--data-dir=\${ETCD_DATA_DIR} \
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=\${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

此时可以先执行该脚本，上述脚本中给出了执行该脚本的example，会进入阻塞等待节点加入状态，如果在一定时间内没有节点加入则会由于timeout（超时）而终止阻塞状态

[root@master01 cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380

可以重新开一个终端查看etcd服务状态

[root@master01 ~]# ps -ef | grep etcd
root      50829  15209  0 20:46 pts/1    00:00:00 /bin/bash ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380
root      50874  50829  0 20:46 pts/1    00:00:00 systemctl restart etcd
root      50880      1  4 20:46 ?        00:00:01 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://192.168.0.128:2380 --listen-client-urls=https://192.168.0.128:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.0.128:2379 --initial-advertise-peer-urls=https://192.168.0.128:2380 --initial-clusteretcd01=https://192.168.0.128:2380,etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
root      50937  50896  0 20:46 pts/2    00:00:00 grep --color=auto etcd

#一定时间内没有节点加入集群就会推出阻塞状态，由于目前其他节点还没有进行任何设置因此一定会退出的
[root@master01 cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380
Job for etcd.service failed because a timeout was exceeded. See "systemctl status etcd.service" and "journalctl -xe" for details.

下面我们将master节点的etcd目录中的文件以及服务启动文件远程拷贝到两个节点服务器的对应目录中，这里以node1节点为例

[root@master01 cfg]# scp -r /opt/etcd [email protected]:/opt
The authenticity of host '192.168.0.129 (192.168.0.129)' can't be established.
ECDSA key fingerprint is SHA256:bkzGRcdP2iJrSTerWtyuqSDENF2mKLWUZHMRkzJZBFI.
ECDSA key fingerprint is MD5:b0:9b:9f:31:de:da:51:8a:d3:ff:87:86:fa:19:63:2c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.129' (ECDSA) to the list of known hosts.
[email protected]'s password: 
etcd.sh                                                                                100% 1812     1.5MB/s   00:00    
etcd                                                                                   100%  509   583.4KB/s   00:00    
etcd                                                                                   100%   18MB 109.8MB/s   00:00    
etcdctl                                                                                100%   15MB 104.3MB/s   00:00    
ca-key.pem                                                                             100% 1679   689.5KB/s   00:00    
ca.pem                                                                                 100% 1265     1.1MB/s   00:00    
server-key.pem                                                                         100% 1675     1.1MB/s   00:00    
server.pem                                                                             100% 1338     2.2MB/s   00:00

[root@master01 cfg]# scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system/
[email protected]'s password: 
etcd.service                                                                           100%  923   430.4KB/s   00:00

下面再到各个节点是进行配置修改（主要是配置文件中的etcd集群节点名称和ip地址）

[root@node01 ~]# cd /opt/etcd/cfg/
[root@node01 cfg]# ls
etcd  etcd.sh
#主要修改配置文件etcd
[root@node01 cfg]# vim etcd
#修改ip地址和节点名称
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.129:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.129:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.129:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.129:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.0.128:2380,etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

此时我们再在master上执行集群搭建命令，然后在两个节点服务器上快速启动etcd服务，服务启动好后会自动退出阻塞状态

[root@master01 cfg]# ./etcd.sh etcd01 192.168.0.128 etcd02=https://192.168.0.129:2380,etcd03=https://192.168.0.130:2380

#两个节点启动etcd服务
[root@node01 cfg]# systemctl start etcd
[root@node02 cfg]# systemctl start etcd

最后我们来检查一下etcd集群状态，如果是health状态则说明etcd集群搭建成功

[root@master01 cfg]# cd ../ssl/ #需要依赖证书来检查该集群状态
[root@master01 ssl]# ls
ca-key.pem  ca.pem  server-key.pem  server.pem
[root@master01 ssl]# /opt/etcd/bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://192.168.0.128:2379,https://192.168.0.129:2379,https://192.168.0.130:2379" cluster-health 
member a25c294d3a391c7c is healthy: got healthy result from https://192.168.0.128:2379
member b2db359ffad36ee5 is healthy: got healthy result from https://192.168.0.129:2379
member eddae83baed564ba is healthy: got healthy result from https://192.168.0.130:2379
cluster is healthy

好了，etcd到此就搭建成功了，后面我们将继续部署其他组件，因此建议如果是实验环境下将虚拟机挂起为最优选择，负责当越来越多的服务来的时候，关机后再启动可能导致各种问题。