kubernetes搭建 二、Etcd集群

etcd需要在master和node上都部署，首先在master上部署，然后把相关的二进制文件和程序拷贝到node上再修改即可

1、下载二进制包然后解压

wget https://github.com/coreos/etcd/releases/download/v3.2.12/etcd-v3.2.12-linux-amd64.tar.gz

2、新建文件夹用来存放k8s相关配置文件和证书，然后把证书文件放到ssl目录下

mkdir -p /opt/kubernetes/{bin,cfg,ssl} 
cp *.pem /opt/kubernetes/ssl

[root@k8s-master-101 kubernetes]# ls /opt/kubernetes/
bin  cfg  ssl

[root@k8s-master-101 ssl]# ls *pem
admin-key.pem  admin.pem  ca-key.pem  ca.pem  kube-proxy-key.pem  kube-proxy.pem  
server-key.pem  server.pem

3、将二进制文件移动到指定目录

mv etcd-v3.2.12-linux-amd64/etcd /opt/kubernetes/bin/
mv etcd-v3.2.12-linux-amd64/etcdctl /opt/kubernetes/bin/

[root@k8s-master-101 bin]# ls etcd*
etcd  etcdctl

4、创建etcd配置文件
vim /opt/kubernetes/cfg/etcd

[root@k8s-master-101 kubernetes]# vim /opt/kubernetes/cfg/etcd
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.101:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.101:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

ETCD_NAME：指定etcd集群名称
ETCD_DATA_DIR：etcd数据目录
ETCD_LISTEN_PEER_URLS：监听的客户端地址
ETCD_LISTEN_CLIENT_URLS：监听的数据端口
ETCD_INITIAL_CLUSTER：集群节点信息
ETCD_INITIAL_CLUSTER_TOKEN：认证的token，可自定义
ETCD_INITIAL_CLUSTER_STATE：集群建立的状态
上边的这个配置文件是master的配置文件内容，配置node节点的配置文件需要修改ETCD_NAME、ETCD_LISTEN_PEER_URLS、ETCD_LISTEN_CLIENT_URLS、ETCD_INITIAL_ADVERTISE_PEER_URLS、ETCD_ADVERTISE_CLIENT_URLS，修改为对应节点的ip地址，还有节点名称要和ETCD_INITIAL_CLUSTER中的对应。

5、创建etcd启动配置文件
vim /usr/lib/systemd/system/etcd.service

[root@k8s-master-101 kubernetes]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/opt/kubernetes/cfg/etcd
ExecStart=/opt/kubernetes/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-state=new \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/ssl/server.pem \
--peer-key-file=/opt/kubernetes/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

6、启动etcd服务

systemctl start etcd
systemctl enable etcd

7、在主节点与从节点间做ssh免密

ssh-keygen
ssh-copy-id [email protected] 
ssh-copy-id [email protected] 

8、将配置文件和证书复制到其他节点上

scp -r /opt/kubernetes/bin/* [email protected]:/opt/kubernetes/bin
scp -r /opt/kubernetes/cfg/* [email protected]:/opt/kubernetes/cfg
scp -r /opt/kubernetes/ssl/* [email protected]:/opt/kubernetes/ssl
scp -r /opt/kubernetes/bin/* [email protected]:/opt/kubernetes/bin
scp -r /opt/kubernetes/cfg/* [email protected]:/opt/kubernetes/cfg
scp -r /opt/kubernetes/ssl/* [email protected]:/opt/kubernetes/ssl

9、把etcd.service也复制过去，因为里面引用的是变量，等等只要修改etcd.conf文件里的变量

scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system
scp /usr/lib/systemd/system/etcd.service [email protected]:/usr/lib/systemd/system

10、修改其他节点的etcd.conf配置文件，改为相应的地址
node1上

[root@k8s-node1-102 cfg]# cat /opt/kubernetes/cfg/etcd 
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.102:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.102:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.102:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.102:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2上

[root@k8s-node2-103 ~]# cat /opt/kubernetes/cfg/etcd
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.103:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.103:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.103:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.101:2380,etcd02=https://10.0.0.102:2380,etcd03=https://10.0.0.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

修改完node1和node2的etcd配置文件，然后启动etcd

systemctl start etcd
systemctl enable etcd

11、在全部节点上，将kubernetes命令路径加入系统变量

echo "PATH=$PATH:/opt/kubernetes/bin" >> /etc/profile
source /etc/profile

12、在随便一台节点进行测试，查看etcd集群状态，使用etcd查询数据都需要使用认证文件，查询格式如下

etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem \
--cert-file=/opt/kubernetes/ssl/server.pem \
--key-file=/opt/kubernetes/ssl/server-key.pem \
cluster-health

[root@k8s-master-101 kubernetes]# etcdctl --ca-file=/opt/kubernetes/ssl/ca.pem --cert-file=/opt/kubernetes/ssl/server.pem --key-file=/opt/kubernetes/ssl/server-key.pem cluster-health
member 1ff1b5249241e477 is healthy: got healthy result from https://10.0.0.103:2379
member 4138ff178a93c6fe is healthy: got healthy result from https://10.0.0.102:2379
member ea5e8a2c2d9c581d is healthy: got healthy result from https://10.0.0.101:2379
cluster is healthy