ETCD集群安装详细步骤

安装注意事项声明：

1.注意事项--环境变量指定了，启动文件的就不要指定了，会报错

etcd 3.4注意事项

ETCD3.4版本ETCDCTL_API=3 etcdctl 和 etcd --enable-v2=false 成为了默认配置，如要使用v2版本，执行etcdctl时候需要设置ETCDCTL_API环境变量，例如：ETCDCTL_API=2 etcdctl
ETCD3.4版本会自动读取环境变量的参数，所以EnvironmentFile文件中有的参数，不需要再次在ExecStart启动参数中添加，二选一，如同时配置，会触发以下类似报错“etcd: conflicting environment variable “ETCD_NAME” is shadowed by corresponding command-line flag (either unset environment variable or disable flag)”
flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API
注意：flannel操作etcd使用的是v2的API，而kubernetes操作etcd使用的v3的API，为了兼容flannel，将默认开启v2版本，故需要配置文件/opt/soft/etcd/etcd.conf中设置 ETCD_ENABLE_V2=“true”
在配合K8S使用的时候，写入的Network网段必须是 /16 段地址，必须与K8S的kube-controller-manager、kube-proxy的cluster-cidr参数值一致（原则上应该反过来说）

另外，flanneld 配置文件中要将集群的地址和证书都配置上，如下示例：

FLANNEL_ETCD_ENDPOINTS="https://192.168.1.54:2379,https://192.168.1.65:2379,https://192.168.1.105:2379"
FLANNEL_ETCD_PREFIX="/flannel/network"
FLANNEL_OPTIONS="-etcd-cafile=/opt/soft/etcd/ssl/ca.pem -etcd-keyfile=/opt/soft/etcd/ssl/server-key.pem  -etcd-certfile=/opt/soft/etcd/ssl/server.pem"

还有flanneld.service的片段：

ExecStart=/opt/soft/flannel/flanneld -ip-masq -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS} $FLANNEL_OPTIONS -etcd-prefix=${FLANNEL_ETCD_PREFIX}

• 

2.etcd启动参数部分不支持域名，否则必须用外部DNS动态解析

3.对等证书

https://blog.csdn.net/xiaozhiit/article/details/108304488

4.动态发现，解决问题

https://www.cnblogs.com/winstom/p/11811373.html#%E9%9D%99%E6%80%81%E9%9B%86%E7%BE%A4

5.集群k8s外部etcd

https://blog.csdn.net/lswzw/article/details/109027255

6.安装ETCD集群

https://blog.csdn.net/eyeofeagle/article/details/101676526

1, 生成证书

#ca机构配置：有效期10年
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

#ca机构配置:　机构名称Comman Name，所在地Country国家, State省, Locality市
cat >  ca-csr.json <<EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Beijing",
            "L": "Beijing"
        }
    ]
}
EOF

#向ca机构申请：证书注册 (中国，北京省，北京市), 提供服务的ip
# Organization Name, Common Name
cat >  server-csr.json <<EOF
{
    "CN": "etcd",
    "hosts": [
    "192.168.56.180",
    "192.168.56.201"
    ], 
    "names": [
        {
            "C": "CN",
            "ST": "BeiJing",
            "L": "BeiJing",
            "O":"aa.com",
            "CN":"beijing.aa.com"
        }
    ]
}
EOF


#下载cfssl工具
curl -s -L -o /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 
curl -s -L -o /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 
curl -s -L -o /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 
chmod +x /bin/cfssl*

#用cfssl工具，生成证书：
mkdir ssl ; cd ssl
cfssl gencert -initca ../ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=../ca-config.json -profile=www ../server-csr.json | cfssljson -bare server
# ls *pem
#ca-key.pem  ca.pem  server-key.pem  server.pem

2, 安装etcd服务

etcd角色名	ip
etcd01	192.168.56.180
etcd02	192.168.56.201

下载etcd：https://github.com/etcd-io/etcd/releases/tag/v3.2.12

#下载etcd　tar包
cat >download-etcd.sh <<EOF
ETCD_VER=v3.2.12
GITHUB_URL=https://github.com/coreos/etcd/releases/download
DOWNLOAD_URL=${GITHUB_URL}
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz  -o  etcd-${ETCD_VER}-linux-amd64.tar.gz
EOF
chmod +x download-etcd.sh
sh download-etcd.sh

#解压etcd　tar包，获取可执行文件：etcd, etcdctl
mkdir /opt/etcd/{bin,cfg,ssl}  -p
tar zxvf etcd-v3.2.12-linux-amd64.tar.gz
mv etcd-v3.2.12-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/
rm -rf  etcd-v3.2.12*
#拷贝证书到etcd服务中
cp * /opt/etcd/ssl/

#编辑etcd配置文件
cat > /opt/etcd/cfg/etcd  <<EOF
#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.56.180:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.56.180:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.56.180:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.56.180:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.56.180:2380,etcd02=https://192.168.56.201:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

#配置systemctl 管理etcd服务
cat  > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure

同步/opt/etcd目录到其他主机

scp -r /opt/etcd  192.168.56.201:/opt/　# /opt/etcd/cfg/etcd 前5行配置, 改为本机ip的配置
scp  /usr/lib/systemd/system/etcd.service   192.168.56.201:/usr/lib/systemd/system/etcd.service  

依次启动所有主机上的etcd服务（第一台机器开始会处于等待状态，知道有新的节点启动，才会正常启动）

systemctl start etcd 
systemctl enable etcd 

ssh-copy-id 192.168.56.201 
ssh 192.168.56.201  "systemctl start etcd "
ssh 192.168.56.201  "systemctl enable etcd "

检查etcd集群：服务状态

#查看系统日志：
[root@jenkins ~]# tailf /var/log/messages
Sep 28 22:25:59 jenkins etcd: 96e8ad8d29182cfd is starting a new election at term 36
Sep 28 22:25:59 jenkins etcd: 96e8ad8d29182cfd became candidate at term 37
Sep 28 22:25:59 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from 96e8ad8d29182cfd at term 37
Sep 28 22:25:59 jenkins etcd: 96e8ad8d29182cfd [logterm: 1, index: 2] sent MsgVote request to bdc976e03235ad9b at term 37
Sep 28 22:26:00 jenkins etcd: 96e8ad8d29182cfd is starting a new election at term 37
Sep 28 22:26:00 jenkins etcd: 96e8ad8d29182cfd became candidate at term 38
Sep 28 22:26:00 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from 96e8ad8d29182cfd at term 38
Sep 28 22:26:00 jenkins etcd: 96e8ad8d29182cfd [logterm: 1, index: 2] sent MsgVote request to bdc976e03235ad9b at term 38
Sep 28 22:26:02 jenkins etcd: 96e8ad8d29182cfd is starting a new election at term 38
Sep 28 22:26:02 jenkins etcd: 96e8ad8d29182cfd became candidate at term 39
Sep 28 22:26:02 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from 96e8ad8d29182cfd at term 39
Sep 28 22:26:02 jenkins etcd: 96e8ad8d29182cfd [logterm: 1, index: 2] sent MsgVote request to bdc976e03235ad9b at term 39
Sep 28 22:26:03 jenkins etcd: 96e8ad8d29182cfd is starting a new election at term 39
Sep 28 22:26:03 jenkins etcd: 96e8ad8d29182cfd became candidate at term 40
Sep 28 22:26:03 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from 96e8ad8d29182cfd at term 40
Sep 28 22:26:03 jenkins etcd: 96e8ad8d29182cfd [logterm: 1, index: 2] sent MsgVote request to bdc976e03235ad9b at term 40
Sep 28 22:26:03 jenkins etcd: health check for peer bdc976e03235ad9b could not connect: dial tcp 192.168.56.201:2380: getsockopt: connection refused
Sep 28 22:26:03 jenkins etcd: peer bdc976e03235ad9b became active
Sep 28 22:26:03 jenkins etcd: established a TCP streaming connection with peer bdc976e03235ad9b (stream Message reader)
Sep 28 22:26:03 jenkins etcd: established a TCP streaming connection with peer bdc976e03235ad9b (stream MsgApp v2 reader)
Sep 28 22:26:03 jenkins etcd: established a TCP streaming connection with peer bdc976e03235ad9b (stream Message writer)
Sep 28 22:26:03 jenkins etcd: established a TCP streaming connection with peer bdc976e03235ad9b (stream MsgApp v2 writer)
Sep 28 22:26:03 jenkins etcd: 96e8ad8d29182cfd [term: 40] ignored a MsgVote message with lower term from bdc976e03235ad9b [term: 2]
Sep 28 22:26:04 jenkins etcd: publish error: etcdserver: request timed out
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd is starting a new election at term 40
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd became candidate at term 41
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from 96e8ad8d29182cfd at term 41
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd [logterm: 1, index: 2] sent MsgVote request to bdc976e03235ad9b at term 41
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd received MsgVoteResp from bdc976e03235ad9b at term 41
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections
Sep 28 22:26:04 jenkins etcd: 96e8ad8d29182cfd became leader at term 41
Sep 28 22:26:04 jenkins etcd: raft.node: 96e8ad8d29182cfd elected leader 96e8ad8d29182cfd at term 41
Sep 28 22:26:04 jenkins etcd: setting up the initial cluster version to 3.2
Sep 28 22:26:04 jenkins etcd: published {Name:etcd01 ClientURLs:[https://192.168.56.180:2379]} to cluster 33793b7f6cbeb4e8
Sep 28 22:26:04 jenkins etcd: ready to serve client requests
Sep 28 22:26:04 jenkins etcd: ready to serve client requests
Sep 28 22:26:04 jenkins etcd: dialing to target with scheme: ""
Sep 28 22:26:04 jenkins etcd: could not get resolver for scheme: ""
Sep 28 22:26:04 jenkins etcd: dialing to target with scheme: ""
Sep 28 22:26:04 jenkins etcd: could not get resolver for scheme: ""
Sep 28 22:26:04 jenkins etcd: serving client requests on 192.168.56.180:2379
Sep 28 22:26:04 jenkins etcd: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
Sep 28 22:26:04 jenkins systemd: Started Etcd Server.
Sep 28 22:26:04 jenkins etcd: set the initial cluster version to 3.2
Sep 28 22:26:04 jenkins etcd: enabled capabilities for version 3.2

#检查服务
[root@jenkins ~]#  /opt/etcd/bin/etcdctl \
 --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
 cluster-health
member 96e8ad8d29182cfd is healthy: got healthy result from https://192.168.56.180:2379
member bdc976e03235ad9b is healthy: got healthy result from https://192.168.56.201:2379