etcd+tls集群部署文档

etcd+tls集群部署文档

前言

kubeadm安装的集群，默认etcd是一个单机的容器化的etcd，这里改造成三节点的etcd集群。

首先我们需要先部署一个三节点的etcd集群，二进制部署，systemd守护进程，并且需要生成ca证书。

ETCD集群详情

kuberntes 系统使用 etcd 存储所有数据，此外calico网络也使用该etcd集群，本文档介绍部署一个三节点高可用 etcd 集群，分别命名为etcd01、etcd02、etcd03

192.168.40.3	etcd01
192.168.40.4	etcd02
192.168.40.5	etcd03

本系列默认使用root用户操作。

创建 CA 证书和秘钥

本文档使用 CloudFlare 的 PKI 工具集 cfssl 来生成 Certificate Authority (CA) 证书和秘钥文件，CA 是自签名的证书，用来签名后续创建的其它 TLS 证书。

安装cfssl

如果不希望将cfssl工具安装到etcd集群节点主机上，可以在其他的主机上进行该操作，生成以后将证书拷贝到部署etcd的节点主机上即可。本文档就是采取这种方法，在一台测试机上执行下面操作。

$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

$ chmod u+x cfssl_linux-amd64

$ mv cfssl_linux-amd64 /usr/local/bin/cfssl



$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

$ chmod u+x cfssljson_linux-amd64

$ mv cfssljson_linux-amd64 /usr/local/bin/cfssljson



$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

$ chmod u+x cfssl-certinfo_linux-amd64

$ mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

生成ETCD的TLS 秘钥和证书

为了保证通信安全，客户端(如 etcdctl) 与 etcd 集群、etcd 集群之间的通信需要使用 TLS 加密。开始创建 etcd TLS 加密所需的证书和私钥。

创建 CA 配置文件

$ cat >  ca-config.json <<EOF

{

  "signing": {

    "default": {

      "expiry": "8760h"

    },

    "profiles": {

      "kubernetes": {

        "usages": [

            "signing",

            "key encipherment",

            "server auth",

            "client auth"

        ],

        "expiry": "8760h"

      }

    }

  }

}

EOF

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个指定的 profile；

signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；

server auth：表示 client 可以用该 CA 对 server 提供的证书进行验证；

client auth：表示 server 可以用该 CA 对 client 提供的证书进行验证；

创建 CA 证书签名请求文件

$ cat >  ca-csr.json <<EOF

{

  "CN": "kubernetes",

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

EOF

"CN"：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；

"O"：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；

生成 CA 证书和私钥

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

$ ls ca*

ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

创建 etcd 证书签名请求

$ cat > etcd-csr.json <<EOF

{

  "CN": "etcd",

  "hosts": [

    "127.0.0.1",

    "192.168.40.3",

    "192.168.40.4",

    "192.168.40.5",

    "192.168.40.24",

    "192.168.40.23",

"192.168.40.22"，

    "etcd01.local.com",

    " etcd02.local.com ",

    " etcd02.local.com ",

    " master01.local.com ",

    " master01.local.com ",

    " master01.local.com ",

  ],

  "key": {

    "algo": "rsa",

    "size": 2048

  },

  "names": [

    {

      "C": "CN",

      "ST": "BeiJing",

      "L": "BeiJing",

      "O": "k8s",

      "OU": "System"

    }

  ]

}

EOF

hosts 字段指定授权使用该证书的 etcd 节点 IP

生成 etcd 证书和私钥

$ cfssl gencert -ca=ca.pem \

  -ca-key=ca-key.pem \

  -config=ca-config.json \

  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

$ ls etcd*

etcd.csr  etcd-csr.json  etcd-key.pem etcd.pem ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

$ rm etcd.csr  etcd-csr.json

将生成好的etcd.pem和etcd-key.pem以及ca.pem三个文件拷贝到目标节点主机的/etc/etcd/ssl目录下(没有就创建)

开始安装etcd
下载二进制文件

$ wget https://etcd.readthedocs.io/en/latest/

$ tar -xvf etcd-v3.3.2-linux-amd64.tar.gz

$ mv etcd-v3.2.11-linux-amd64/etcd* /usr/local/bin/

创建 etcd 的 systemd unit 文件

$ mkdir -p /var/lib/etcd  # 必须先创建工作目录

$ yum -y install ntpdate kernel-devel gcc-c++

$ntpdate ntp.pool.org

$ vi install_etcd.sh

#!/bin/bash

# Copyright 2018 The Kubernetes Authors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.



## Create etcd.conf, etcd.service, and start etcd service.





mkdir -p /usr/local/kubernetes/config

mkdir -p /usr/local/kubernetes/bin



etcd_data_dir=/var/lib/etcd

mkdir -p ${etcd_data_dir}



#ETCD_NAME=${1:-"etcd01"}

#ETCD_INITIAL_CLUSTER=${3:-"etcd01=https://192.168.40.3:2380,etcd02=https://192.168.40.4:2380,etcd03=https://192.168.40.5:2380"}

#CURRENT_HOST_IP=`ip a | grep ens32 | grep 'inet ' | awk '{ print $2}'`



ETCD_NAME="$1"

ETCD_INITIAL_CLUSTER="$2"



if [ ! $ETCD_NAME ]; then

  echo "ENTER ETCD_NAME eg:etcd01"

  exit 1

fi



if [ ! $ETCD_INITIAL_CLUSTER ]; then

  echo "ENTER ETCD_INITIAL_CLUSTER eg:etcd01=https://192.168.40.3:2380,etcd02=https://192.168.40.4:2380,etcd03=https://192.168.40.5:2380"

  exit 1

fi



cp -rf bin/* /usr/local/kubernetes/bin

chmod +x /usr/local/kubernetes/bin/*



ETCD_LISTEN_IP=`ip a | grep ens32 | grep 'inet ' | awk '{ print $2}' | awk -F / '{print $1}'`

ETCD_DATA_DIR="${etcd_data_dir}/default.etcd"

ETCD_LISTEN_PEER_URLS="https:// ${ETCD_LISTEN_IP}:2380"

ETCD_LISTEN_CLIENT_URLS="https://${ETCD_LISTEN_IP}:2379"

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_LISTEN_IP}:2380"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_LISTEN_IP}:2379"

CERT_FILE=/etc/etcd/ssl/etcd.pem

KEY_FILE=/etc/etcd/ssl/etcd-key.pem

PERR_CERT_FILE=/etc/etcd/ssl/etcd.pem

PERR_KEY_FILE=/etc/etcd/ssl/etcd-key.pem

TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem

PERR_TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem

cat <<EOF >/usr/local/kubernetes/config/etcd.conf

# [member]

ETCD_NAME="${ETCD_NAME}"

ETCD_DATA_DIR="${etcd_data_dir}/default.etcd"

ETCD_LISTEN_PEER_URLS="https://192.168.40.3:2380"

ETCD_LISTEN_CLIENT_URLS="https://192.168.40.3:2379"



#[cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_LISTEN_IP}:2380"

ETCD_INITIAL_CLUSTER="${ETCD_INITIAL_CLUSTER}"

ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"

ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_LISTEN_IP}:2379"



#[certs]

CERT_FILE=/etc/etcd/ssl/etcd.pem

KEY_FILE=/etc/etcd/ssl/etcd-key.pem

PERR_CERT_FILE=/etc/etcd/ssl/etcd.pem

PERR_KEY_FILE=/etc/etcd/ssl/etcd-key.pem

TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem

PERR_TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem

EOF



cat <<EOF >/usr/lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target



[Service]

Type=notify

WorkingDirectory=${etcd_data_dir}

EnvironmentFile=-/usr/local/kubernetes/config/etcd.conf

ExecStart=/usr/local/kubernetes/bin/etcd \\

        --name=${ETCD_NAME} \\

        --data-dir=${ETCD_DATA_DIR} \\

        --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \\

        --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS} \\

        --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \\

        --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \\

        --initial-cluster=${ETCD_INITIAL_CLUSTER} \\

        --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \\

        --initial-cluster-state=${ETCD_INITIAL_CLUSTER_STATE} \\

        --cert-file=${CERT_FILE} \\

        --key-file=${KEY_FILE} \\

        --peer-cert-file=${PERR_CERT_FILE} \\

        --peer-key-file=${PERR_KEY_FILE} \\

        --trusted-ca-file=${TRUSTED_CA_FILE} \\

        --peer-trusted-ca-file=${PERR_TRUSTED_CA_FILE} \\

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF



# systemctl daemon-reload

# systemctl enable etcd

# systemctl restart etcd

指定 etcd 的工作目录和数据目录为 /var/lib/etcd，需在启动服务前创建这个目录；

为了保证通信安全，需要指定 etcd 的公私钥(cert-file和key-file)、Peers 通信的公私钥和 CA 证书(peer-cert-file、peer-key-file、peer-trusted-ca-file)、客户端的CA证书（trusted-ca-file）；

--initial-cluster-state 值为 new 时，--name 的参数值必须位于 --initial-cluster 列表中；

启动 etcd 服务

$ mv etcd.service /etc/systemd/system/

$ systemctl daemon-reload

$ systemctl enable etcd

$ systemctl start etcd

$ systemctl status etcd

$./install_etcd.sh etcd01 etcd01=https://192.168.40.3:2380,etcd02=https://192.168.40.4:2380,etcd03=https://192.168.40.5:2380

$./install_etcd.sh etcd02 etcd01=https://192.168.40.3:2380,etcd02=https://192.168.40.4:2380,etcd03=https://192.168.40.5:2380

$./install_etcd.sh etcd03 etcd01=https://192.168.40.3:2380,etcd02=https://192.168.40.4:2380,etcd03=https://192.168.40.5:2380

验证服务

部署完 etcd 集群后，在任意一台 etcd 集群节点上执行如下命令：

$ etcdctl \

  --endpoints=https://192.168.40.3:2379  \

  --ca-file=/etc/etcd/ssl/ca.pem \

  --cert-file=/etc/etcd/ssl/etcd.pem \

  --key-file=/etc/etcd/ssl/etcd-key.pem \

  cluster-health
三台 etcd 的输出均为 healthy 时表示集群服务正常

member 71df888fdf6f0bb9 is healthy: got healthy result from https://192.168.40.3:2379

member 73b5207bc2491164 is healthy: got healthy result from https://192.168.40.4:2379

member 7a4ddb7c77253f4b is healthy: got healthy result from https://192.168.40.5:2379

etcd+tls集群部署文档

猜你喜欢