k8s集群 添加节点过程记录及问题解决。

环境准备

1.关闭防火墙，关闭selinux(生产环境按需关闭或打开)

systemctl disable firewalld.service
systemctl stop firewalld.service
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

检查：

systemctl is-enabled firewalld.service
systemctl status firewalld.service
getenforce

2.同步服务器时间，选择公网ntpd服务器或者自建ntpd服务器

3.关闭swap分区

echo "vm.swappiness=1">>/etc/sysctl.conf
sysctl -p
**检查：**
sysctl -a|grep "vm.swappiness"

4.集群所有节点主机可以相互解析
5.master对node节点ssh互信

ssh-keygen -t rsa
ssh-copy-id -i /root/.ssh/id_rsa.pub 172.16.0.95

6.配置系统内核参数使流过网桥的流量也进入iptables/netfilter框架

  modprobe br_netfilter  
  echo -e 'net.bridge.bridge-nf-call-iptables = 1 \nnet.bridge.bridge-nf-call-ip6tables = 1' >> /etc/sysctl.conf  && sysctl -p

或者

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

7.修改主机名

hostnamectl set-hostname master1
echo 'master1'>/etc/hostname

安装docker kubeadm kubectl kubelet kubernetes-cni

1: 配置yum（所有节点）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2: 安装kubeadm和docker

yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum -y install docker-ce docker-ce-selinux 
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet
systemctl enable docker && systemctl start docker

3: 下载镜像：

#!/bin/bash

images=(
    kube-apiserver:v1.14.2
    kube-controller-manager:v1.14.2
    kube-scheduler:v1.14.2
    kube-proxy:v1.14.2
    kube-apiserver:v1.14.2
    pause:3.1
    etcd:3.3.10
    coredns:1.3.1
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done

4: 加入集群

kubeadm join 172.16.0.92:6443 --token 769elv.w7ndytgXXXXXXX \
    --discovery-token-ca-cert-hash sha256:685b7b8cb7ca0a0e3b65f3b68433e4d67f8927b54c5beXXXXXX

kubeadm join报错及解决

1、报错：detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”

> kubeadm join ---
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/

原因：centos7的cgroup driver为systemd，docker默认的cgroup driver为cgroupfs，使用两种cgroup driver控制资源的话会导致资源分配不均。
解决方法：修改docker的cgroup driver为systemd

# Install Docker CE
## Set up the repository
### Install required packages.
yum install yum-utils device-mapper-persistent-data lvm2

### Add Docker repository.
yum-config-manager \
  --add-repo \
  https://download.docker.com/linux/centos/docker-ce.repo

## Install Docker CE.
yum update && yum install docker-ce-18.06.2.ce

## Create /etc/docker directory.
mkdir /etc/docker

# Setup daemon.
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# Restart Docker
systemctl daemon-reload
systemctl restart docker

2、报错：
error execution phase preflight: couldn’t validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s

> kubeadm join ---
error execution phase preflight: couldn't validate the identity of the API Server: abort connecting to API servers after timeout of 5m0s

原因：master节点的token过期了
解决：创建新的token

#得到token
>kubeadm token create 
#得到discovery-token-ca-cert-hash
> openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'  

重新添加就可以join成功啦。