环境
三台主机,一台master,两台node
192.168.31.11 k8s-1 作为master 192.168.31.12 k8s-2 作为node节点 192.168.31.13 k8s-3 作为node节点
每台主机Centos版本使用 CentOS Linux release 7.6.1810 (Core)
软件版本信息:
docker-ce-selinux-17.03.3.ce-1.el7.noarch docker-ce-17.03.2.ce-1.el7.centos.x86_64 kubelet-1.11.1-0.x86_64 kubeadm-1.11.1-0.x86_64 kubernetes-cni-0.6.0-0.x86_64 kubectl-1.11.1-0.x86_64
1.主机的基本设置
修改主机名
# 192.168.31.11 hostnamectl set-hostname k8s-1 # 192.168.31.12 hostnamectl set-hostname k8s-2 # 192.168.31.13 hostnamectl set-hostname k8s-3
修改每台主机的hosts文件添加如下内容
192.168.31.11 k8s-1 192.168.31.12 k8s-2 192.168.31.13 k8s-3
每台主机都系统时间同步
# 修改时区 ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime yum -y install ntpdate ntpdate cn.pool.ntp.org
每台主机都关闭防火墙及selinux
systemctl disable firewalld systemctl stop firewalld setenforce 0 # 修改文件设置selinux永久关闭 vi /etc/sysconfig/selinux SELINUX=enforcing
关闭swap分区(如果创建主机时删除swap分区,这里不用修改)
swapoff -a # 修改文件防止下次启动挂载swap sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
2.配置安装前环境
以下操作在每台主机上都需要操作
每台主机都设置docker-ce源信息
# 下载wget yum -y install wget # 下载源repo wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo # 修改源到 sed -i '[email protected]@mirrors.tuna.tsinghua.edu.cn/docker-ce@g' /etc/yum.repos.d/docker-ce.repo
每台主机都设置Kubernetes仓库
# vi /etc/yum.repos.d/kubernetes.repo 添加如下内容 [kubernetes] name=Kubernetes Repo baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=0 enable=1
更新yum仓库
yum clean all
yum repolist
每台主机都需要安装docker-ce包(master上面有些服务是需要运行在docker容器中)
# 需要先升级docker-ce-selinux,版本太低 yum -y install https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/7/x86_64/stable/Packages/docker-ce-selinux-17.03.3.ce-1.el7.noarch.rpm # 安装docker-ce yum -y install docker-ce-17.03.2.ce
设置并启动docker服务
# 配置加速器到配置文件 mkdir -p /etc/docker tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://registry.docker-cn.com"] } EOF # 启动服务 systemctl daemon-reload systemctl enable docker systemctl start docker # 打开iptables内生的桥接相关功能 echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
3.master上面安装并启动服务
安装需要的rpm包
# 先安装 kubelet-1.10.10 # 原因是因为直接安装kubelet1.11.1的话会导致kubernetes-cni版本过高, # 这里安装kubelet1.10.10会依赖安装kubernetes-cni-0.6.0版本 # 暂时没有找到好的办法解决这个问题 yum -y install kubelet-1.10.10 # 安装需要的版本1.11.10 yum -y install kubeadm-1.11.1 kubelet-1.11.1 kubectl-1.11.1
配置启动kubelet服务
vi /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS="--fail-swap-on=false" KUBE_PROXY=MODE=ipvs # 配置为开机启动 systemctl enable kubelet
提前拉取镜像(kubeadm在初始化过程会拉取镜像,但是镜像都是Google提供,无法下载)
docker pull xiyangxixia/k8s-proxy-amd64:v1.11.1 docker tag xiyangxixia/k8s-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1 docker pull xiyangxixia/k8s-scheduler:v1.11.1 docker tag xiyangxixia/k8s-scheduler:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1 docker pull xiyangxixia/k8s-controller-manager:v1.11.1 docker tag xiyangxixia/k8s-controller-manager:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1 docker pull xiyangxixia/k8s-apiserver-amd64:v1.11.1 docker tag xiyangxixia/k8s-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1 docker pull xiyangxixia/k8s-etcd:3.2.18 docker tag xiyangxixia/k8s-etcd:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18 docker pull xiyangxixia/k8s-coredns:1.1.3 docker tag xiyangxixia/k8s-coredns:1.1.3 k8s.gcr.io/coredns:1.1.3 docker pull xiyangxixia/k8s-pause:3.1 docker tag xiyangxixia/k8s-pause:3.1 k8s.gcr.io/pause:3.1 docker pull xiyangxixia/k8s-flannel:v0.10.0-s390x docker tag xiyangxixia/k8s-flannel:v0.10.0-s390x quay.io/coreos/flannel:v0.10.0-s390x docker pull xiyangxixia/k8s-flannel:v0.10.0-ppc64le docker tag xiyangxixia/k8s-flannel:v0.10.0-ppc64le quay.io/coreos/flannel:v0.10.0-ppc64l docker pull xiyangxixia/k8s-flannel:v0.10.0-arm docker tag xiyangxixia/k8s-flannel:v0.10.0-arm quay.io/coreos/flannel:v0.10.0-arm docker pull xiyangxixia/k8s-flannel:v0.10.0-amd64 docker tag xiyangxixia/k8s-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
初始化Kubernetes master节点
kubeadm init --kubernetes-version=v1.11.1 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap # --kubernetes-version=v1.11.1 因为无法连接Google服务所以需要手动指定版本信息 # --pod-network-cidr=10.244.0.0/16 pod获取的ip的网段,默认即可 # --service-cidr=10.96.0.0/12 指定service网段,不要和物理服务节点同一个网段 # --ignore-preflight-errors=Swap/all 忽略错误
初始化成功后,根据提示信息执行操作
mkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config
等待所有服务启动(docker中运行)后,使用命令查询信息
# 查询组件信息 kubectl get cs # 查询节点信息(flannel还没有部署,提示为NotReady) kubectl get nodes # 查询命名空间 kubectl get ns
部署网络插件flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml # 查看下载的images docker images | grep flannel
再次验证节点信息,显示Ready
kubectl get nodes
4.节点node上安装并启动服务
两个节点操作完全相同
安装需要的rpm包,确保防火墙,selinux关闭并设置永久关闭,docker-ce安装并启动
# 先安装 kubelet-1.10.10 # 原因是因为直接安装kubelet1.11.1的话会导致kubernetes-cni版本过高, # 这里安装kubelet1.10.10会依赖安装kubernetes-cni-0.6.0版本 # 暂时没有找到好的办法解决这个问题 yum -y install kubelet-1.10.10 # kubectl 其实没有必要安装 yum -y install kubeadm-1.11.1 kubelet-1.11.1 kubectl-1.11.1
在master上面操作,创建token
# 查看当前所有的token kubeadm token list # 创建token,并记下 kubeadm token create [root@k8s-1 .kube]# kubeadm token create 8d5cbr.n84orohakj3o5ppd # 如果没有值--discovery-token-ca-cert-hash，可以通过在master节点上运行以下命令链来获取 openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //' [root@k8s-1 .kube]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ > openssl dgst -sha256 -hex | sed 's/^.* //' febac84e25f527f8ee8770a35165164ea8f930929ae0d648405240b3850f5c53
提前下载镜像(初始化过程中会拉取镜像,但是下载不了)
docker pull xiyangxixia/k8s-pause:3.1 docker tag xiyangxixia/k8s-pause:3.1 k8s.gcr.io/pause:3.1 docker pull xiyangxixia/k8s-proxy-amd64:v1.11.1 docker tag xiyangxixia/k8s-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1 docker pull xiyangxixia/k8s-flannel:v0.10.0-amd64 docker tag xiyangxixia/k8s-flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
根据创建的token初始化node并加入集群
kubeadm join 192.168.31.11:6443 --token 8d5cbr.n84orohakj3o5ppd --discovery-token-ca-cert-hash sha256:febac84e25f527f8ee8770a35165164ea8f930929ae0d648405240b3850f5c53 --ignore-preflight-errors=Swap # 192.168.31.11:6443 这里是master的ip地址,master必须关闭防火墙,关闭selinux # --token 这里是kubeadm token create 输入的值 # --discovery-token-ca-cert-hash sha256: 这里是执行命令获取的sha256的值
5.验证集群是否初始化成功
查询节点镜像是否下载
[root@k8s-1 .kube]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 2 months ago 52.5 MB k8s.gcr.io/kube-proxy-amd64 v1.11.1 d5c25579d0ff 8 months ago 97.8 MB xiyangxixia/k8s-proxy-amd64 v1.11.1 d5c25579d0ff 8 months ago 97.8 MB k8s.gcr.io/kube-apiserver-amd64 v1.11.1 816332bd9d11 8 months ago 187 MB xiyangxixia/k8s-apiserver-amd64 v1.11.1 816332bd9d11 8 months ago 187 MB k8s.gcr.io/kube-controller-manager-amd64 v1.11.1 52096ee87d0e 8 months ago 155 MB xiyangxixia/k8s-controller-manager v1.11.1 52096ee87d0e 8 months ago 155 MB k8s.gcr.io/kube-scheduler-amd64 v1.11.1 272b3a60cd68 8 months ago 56.8 MB xiyangxixia/k8s-scheduler v1.11.1 272b3a60cd68 8 months ago 56.8 MB xiyangxixia/k8s-coredns 1.1.3 b3b94275d97c 10 months ago 45.6 MB k8s.gcr.io/coredns 1.1.3 b3b94275d97c 10 months ago 45.6 MB k8s.gcr.io/etcd-amd64 3.2.18 b8df3b177be2 12 months ago 219 MB xiyangxixia/k8s-etcd 3.2.18 b8df3b177be2 12 months ago 219 MB quay.io/coreos/flannel v0.10.0-s390x 463654e4ed2d 14 months ago 47 MB xiyangxixia/k8s-flannel v0.10.0-s390x 463654e4ed2d 14 months ago 47 MB quay.io/coreos/flannel v0.10.0-ppc64l e2f67d69dd84 14 months ago 53.5 MB xiyangxixia/k8s-flannel v0.10.0-ppc64le e2f67d69dd84 14 months ago 53.5 MB xiyangxixia/k8s-flannel v0.10.0-arm c663d02f7966 14 months ago 39.9 MB quay.io/coreos/flannel v0.10.0-arm c663d02f7966 14 months ago 39.9 MB quay.io/coreos/flannel v0.10.0-amd64 f0fad859c909 14 months ago 44.6 MB xiyangxixia/k8s-flannel v0.10.0-amd64 f0fad859c909 14 months ago 44.6 MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 15 months ago 742 kB xiyangxixia/k8s-pause 3.1 da86e6ba6ca1 15 months ago 742 kB
查询节点信息(master上执行)是否都是Ready
[root@k8s-1 .kube]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-1 Ready master 14m v1.11.1 k8s-2 Ready <none> 4m v1.11.1 k8s-3 Ready <none> 4m v1.11.1
查询kube-system命名空间下的关于node节点的pod信息
[root@k8s-1 .kube]# kubectl get pods -n kube-system -o wide NAME READY STATUS RESTARTS AGE IP NODE coredns-78fcdf6894-44qf5 1/1 Running 0 14m 10.244.0.2 k8s-1 coredns-78fcdf6894-bxb2m 1/1 Running 0 14m 10.244.0.3 k8s-1 etcd-k8s-1 1/1 Running 0 13m 192.168.31.11 k8s-1 kube-apiserver-k8s-1 1/1 Running 0 13m 192.168.31.11 k8s-1 kube-controller-manager-k8s-1 1/1 Running 0 14m 192.168.31.11 k8s-1 kube-flannel-ds-amd64-cr8j8 1/1 Running 0 6m 192.168.31.11 k8s-1 kube-flannel-ds-amd64-kxk5w 1/1 Running 0 4m 192.168.31.12 k8s-2 kube-flannel-ds-amd64-pk4zl 1/1 Running 0 4m 192.168.31.13 k8s-3 kube-proxy-mxsrg 1/1 Running 0 4m 192.168.31.12 k8s-2 kube-proxy-tp95q 1/1 Running 0 4m 192.168.31.13 k8s-3 kube-proxy-twpvt 1/1 Running 0 14m 192.168.31.11 k8s-1 kube-scheduler-k8s-1 1/1 Running 0 14m 192.168.31.11 k8s-1