一、以http方式搭建etcd集群
1、准备好三台机器,三台机器如下：
IP地址 主机名称 安装服务
172.31.72.142 master1 Etcd、Maser节点
172.31.82.187 master2 Etcd、Node节点
172.31.11.86 master3 Etcd、Node节点
在172.31.72.142机器，执行hostnamectl set-hostname master1
在172.31.82.187 机器，执行hostnamectl set-hostname master2
在172.31.82.187 机器，执行hostnamectl set-hostname master3
在三台机器，vim /etc/hosts 添加master1,master2,master3
[root@master1 ssl]# cat /etc/hosts
172.31.72.142 master1
172.31.82.187 master2
172.31.11.86 master3
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
2、防火墙打开端口2380，2379。
在3台主机都执行以下命令：
# cat /etc/sysconfig/iptables
Generated by iptables-save v1.4.21 on Thu Jul 5 07:05:39 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1654:201815]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# systemctl reload iptables
# systemctl restart iptables
工作中，由于etcd都是内网访问，所以都是关闭防火墙，关闭iptables，利用AWS安全组，控制对外访问端口。
停掉iptables :
systemctl stop iptables
systemctl disable iptables
3、在三台机器上，分别安装etcd，分别执行yum install -y etcd。我这里安装的版本etcd-3.2.18
4、centos7安装etcd的，默认配置文件路径。/etc/etcd/etcd.conf。在三台主机分别执行 vim /etc/etcd/etcd.conf，配置文件修改点如下：
master1(72.31.72.142):
#[Member]
ETCD_DATA_DIR=“/app/etcd"
ETCD_LISTEN_PEER_URLS="http://172.31.72.142:2380,http://127.0.0.1:2380”
ETCD_LISTEN_CLIENT_URLS="http://172.31.72.142:2379,http://127.0.0.1:2379”
ETCD_NAME="etcd1"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.72.142:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.31.72.142:2379”
ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"
ETCD_INITIAL_CLUSTER_STATE="new"
master2(172.31.82.187):
#[Member]
ETCD_DATA_DIR=“/app/etcd”
ETCD_LISTEN_PEER_URLS="http://172.31.82.187:2380,http://127.0.0.1:2380”
ETCD_LISTEN_CLIENT_URLS="http://172.31.82.187:2379,http://127.0.0.1:2379”
ETCD_NAME="etcd2"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.82.187:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.31.82.187:2379”
ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"
ETCD_INITIAL_CLUSTER_STATE="new”
master3(172.31.11.86 ):
#[Member]
ETCD_DATA_DIR=“/app/etcd”
ETCD_LISTEN_PEER_URLS="http://172.31.11.86:2380,http://127.0.0.1:2380”
ETCD_LISTEN_CLIENT_URLS="http://172.31.11.86:2379,http://127.0.0.1:2379”
ETCD_NAME="etcd3"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.11.86:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://172.31.11.86:2379”
ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"
ETCD_INITIAL_CLUSTER_STATE="new”
5、由于yum安装的etcd默认启动，是以etcd用户启动，可能会出现ETCD_DATA_DIR目录权限问题，这里，直接修改三台机器，都以root身份启动，
vim /lib/systemd/system/etcd.service
[Service]
User=etcd
将User=etcd修改为User=root
6、重新加载配置文件。
三台机器执行以下命令：
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
7、验证一下etcd集群
在随便一台机器上执行以下命令
ETCDCTL_API=3 etcdctl member list
如果出现，三台机器列表，代表集群(http方式）搭建成功。
二、搭建SSL etcd集群
在master1机器执行以下步骤：
1、下载cfssl工具，并按照cfssl
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
2、设置/usr/local/bin可以直接加载到用户环境中，这里我用的AWS，需要设置一下才能把/usr/local/bin路径加载到用户环境。
在/etc/profile文件末尾，添加如下：
PATH="$PATH:/usr/local/bin"
export PATH
添加完成，执行source /etc/profile 重载当前用户环境
3、生成证书
#cd /etc/etcd
#mkdir ssl
#cd ssl
#touch build-server.sh
#vim build-server.sh
echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -
echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json
export ADDRESS=172.31.72.142,172.31.82.187,172.31.11.86,master1,master2,master3
export NAME=kubernetes
echo '{"CN":"'$NAME'","hosts":["localhost","127.0.0.1","0.0.0.0","172.31.72.142","172.31.82.187","172.31.11.86","master1","master2","master3"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
export ADDRESS=
export NAME=client
echo '{"CN":"'$NAME'","hosts":["localhost","127.0.0.1","0.0.0.0","172.31.72.142","172.31.82.187","172.31.11.86","master1","master2","master3"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME
#chmod +x build-server.sh
#sh build-server.sh
4、将ssl目录复制到master2，master3机器上。我这里用的AWS,执行以下命令，上传ssl目录到master1，master2机器。
scp -i /home/centos/aws.pem -rp /etc/etcd/ssl centos@master2:/home/centos
scp -i /home/centos/aws.pem -rp /etc/etcd/ssl centos@master3:/home/centos
5、配置etcd启动证书
master1(172.31.72.142)
[root@master1 ssl]# vim /lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=root
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd1 \
--data-dir=/data/etcd \
--listen-client-urls https://172.31.72.142:2379,https://127.0.0.1:2379 \
--advertise-client-urls https://172.31.72.142:2379,https://127.0.0.1:2379 \
--listen-peer-urls https://172.31.72.142:2380 \
--initial-advertise-peer-urls https://172.31.72.142:2380 \
--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \
--initial-cluster-token etcd-cluster-token \
--initial-cluster-state new \
--cert-file=/etc/etcd/ssl/kubernetes.pem \
--key-file=/etc/etcd/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \
--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-client-cert-auth=true \
--client-cert-auth=true"
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
master2(172.31.82.187)
[root@master2 etcd] cd /etc/etcd
[root@master2 etcd] cp -rf /home/centos/ssl .
[root@master2 etcd] vim /lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=root
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd2 \
--data-dir=/data/etcd \
--listen-client-urls https://172.31.82.187:2379,https://127.0.0.1:2379 \
--advertise-client-urls https://172.31.82.187:2379,https://127.0.0.1:2379 \
--listen-peer-urls https://172.31.82.187:2380 \
--initial-advertise-peer-urls https://172.31.82.187:2380 \
--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \
--initial-cluster-token etcd-cluster-token \
--initial-cluster-state new \
--cert-file=/etc/etcd/ssl/kubernetes.pem \
--key-file=/etc/etcd/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \
--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-client-cert-auth=true \
--client-cert-auth=true"
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
master3(172.31.11.86)
[root@master2 etcd] cd /etc/etcd
[root@master2 etcd] cp -rf /home/centos/ssl .
[root@master2 etcd] vim /lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=root
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd3 \
--data-dir=/data/etcd \
--listen-client-urls https://172.31.11.86:2379,https://127.0.0.1:2379 \
--advertise-client-urls https://172.31.11.86:2379,https://127.0.0.1:2379 \
--listen-peer-urls https://172.31.11.86:2380 \
--initial-advertise-peer-urls https://172.31.11.86:2380 \
--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \
--initial-cluster-token etcd-cluster-token \
--initial-cluster-state new \
--cert-file=/etc/etcd/ssl/kubernetes.pem \
--key-file=/etc/etcd/ssl/kubernetes-key.pem \
--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \
--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-client-cert-auth=true \
--client-cert-auth=true"
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
6、在master1，master2，master3执行，systemctl daemon-reload。
如果按照之前http的方式搭建etcd集群，这次重新启动etcd，如果出现，cluster id不匹配的话，请执行以下命令：
# systemctl stop etcd
# rm -rf /etc/data
# systemctl start etcd
7、测试SSL etcd集群安装情况，在任何一台机器测试，如下所示：
[root@master2 ssl]# ETCDCTL_API=3 etcdctl --write-out=table \
--cert=/etc/etcd/ssl/client.pem \
--key=/etc/etcd/ssl/client-key.pem \
--cacert=/etc/etcd/ssl/ca.pem \
--endpoints=https://master1:2379,https://master2:2379,https://master3:2379 \
member list
+------------------+---------+-------+----------------------------+---------------------------------------------------+
| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS |
+------------------+---------+-------+----------------------------+---------------------------------------------------+
| 1fdc3a77c4d295e9 | started | etcd2 | https://172.31.82.187:2380 | https://127.0.0.1:2379,https://172.31.82.187:2379 |
| 3c7af0c898334bb0 | started | etcd1 | https://172.31.72.142:2380 | https://127.0.0.1:2379,https://172.31.72.142:2379 |
| 9089a7ab14781fca | started | etcd3 | https://172.31.11.86:2380 | https://127.0.0.1:2379,https://172.31.11.86:2379 |
+------------------+---------+-------+----------------------------+---------------------------------------------------+
[root@master2 ssl]#