ubuntu 搭建k8s集群中篇

概述

在本章中将会使用kubeadm 来搭建k8s集群。

一、安装kubeadm

1 关闭防火墙和关闭swap

• 关闭防火墙

  ufw disable
  

• 关闭swap

  swapoff -a
  
  # 注释 swap 开头的行 避免开机启动
  vi /etc/fstab
  

2 配置软件源

2.1 安装系统工具

	apt-get update && apt-get install -y apt-transport-https

2.2 安装 GPG 证书

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

2.3 写入软件源

注意：我们用系统代号为 bionic，但目前阿里云不支持，所以沿用 16.04 的 xenial

cat << EOF >/etc/apt/sources.list.d/kubernetes.list
	> deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
	> EOF

3 安装 kubeadm，kubelet，kubectl

3.1 安装

	apt-get update  
	apt-get install -y kubelet kubeadm kubectl

3.2 安装过程如下，注意 kubeadm 的版本号

Reading package lists... Done
	Building dependency tree       
	Reading state information... Done
	The following additional packages will be installed:
	  conntrack cri-tools kubernetes-cni socat
	The following NEW packages will be installed:
	  conntrack cri-tools kubeadm kubectl kubelet kubernetes-cni socat
	0 upgraded, 7 newly installed, 0 to remove and 96 not upgraded.
	Need to get 50.6 MB of archives.
	After this operation, 290 MB of additional disk space will be used.
	Get:1 http://mirrors.aliyun.com/ubuntu bionic/main amd64 conntrack amd64 1:1.4.4+snapshot20161117-6ubuntu2 [30.6 kB]
	Get:2 http://mirrors.aliyun.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB]
	Get:3 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-tools amd64 1.12.0-00 [5,343 kB]
	Get:4 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.7.5-00 [6,473 kB]
	Get:5 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubelet amd64 1.14.1-00 [21.5 MB]
	Get:6 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.14.1-00 [8,806 kB]
	Get:7 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubeadm amd64 1.14.1-00 [8,150 kB]
	Fetched 50.6 MB in 5s (9,912 kB/s) 
	Selecting previously unselected package conntrack.
	(Reading database ... 67205 files and directories currently installed.)
	Preparing to unpack .../0-conntrack_1%3a1.4.4+snapshot20161117-6ubuntu2_amd64.deb ...
	Unpacking conntrack (1:1.4.4+snapshot20161117-6ubuntu2) ...
	Selecting previously unselected package cri-tools.
	Preparing to unpack .../1-cri-tools_1.12.0-00_amd64.deb ...
	Unpacking cri-tools (1.12.0-00) ...
	Selecting previously unselected package kubernetes-cni.
	Preparing to unpack .../2-kubernetes-cni_0.7.5-00_amd64.deb ...
	Unpacking kubernetes-cni (0.7.5-00) ...
	Selecting previously unselected package socat.
	Preparing to unpack .../3-socat_1.7.3.2-2ubuntu2_amd64.deb ...
	Unpacking socat (1.7.3.2-2ubuntu2) ...
	Selecting previously unselected package kubelet.
	Preparing to unpack .../4-kubelet_1.14.1-00_amd64.deb ...
	Unpacking kubelet (1.14.1-00) ...
	Selecting previously unselected package kubectl.
	Preparing to unpack .../5-kubectl_1.14.1-00_amd64.deb ...
	Unpacking kubectl (1.14.1-00) ...
	Selecting previously unselected package kubeadm.
	Preparing to unpack .../6-kubeadm_1.14.1-00_amd64.deb ...
	Unpacking kubeadm (1.14.1-00) ...
	Setting up conntrack (1:1.4.4+snapshot20161117-6ubuntu2) ...
	Setting up kubernetes-cni (0.7.5-00) ...
	Setting up cri-tools (1.12.0-00) ...
	Setting up socat (1.7.3.2-2ubuntu2) ...
	Setting up kubelet (1.14.1-00) ...
	Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service → /lib/systemd/system/kubelet.service.
	Setting up kubectl (1.14.1-00) ...
	Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
	# 注意这里的版本号，我们使用的是 kubernetes v1.14.1
	Setting up kubeadm (1.14.1-00) ...

3.3 设置 kubelet 自启动，并启动 kubelet

	systemctl enable kubelet && systemctl start kubelet

相关组件解释

• kubeadm：用于初始化 Kubernetes 集群
• kubectl：Kubernetes 的命令行工具，主要作用是部署和管理应用，查看各种资源，创建，删除和更新组件
• kubelet：主要负责启动 Pod 和容器

二、配置 kubeadm 并初始化主节点

1 配置kubeamd

1.1 修改主节点 配置信息

导出配置文件

kubeadm config print init-defaults --kubeconfig ClusterConfiguration > kubeadm.yml

1.2 修改配置为如下内容

需要修改的内容如下

apiVersion: kubeadm.k8s.io/v1beta1
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  # 修改为主节点 IP
  advertiseAddress: 192.168.141.130
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: kubernetes-master
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
# 国内不能访问 Google，修改为阿里云
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
# 修改版本号
kubernetesVersion: v1.14.1
networking:
  dnsDomain: cluster.local
  # 配置成 Calico 的默认网段
  podSubnet: "192.168.0.0/16"
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
# 开启 IPVS 模式
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs

1.3 查看和拉取镜像

• 查看所需镜像列表

kubeadm config images list --config kubeadm.yml

• 拉取镜像

kubeadm config images pull --config kubeadm.yml

2 初始化主节点

2.1 安装 kubernetes 主节点

执行以下命令初始化主节点，该命令指定了初始化时需要使用的配置文件，其中添加 --experimental-upload-certs 参数可以在后续执行加入节点时自动分发证书文件。追加的 tee kubeadm-init.log 用以输出日志。

初始化命令如下

kubeadm init --config=kubeadm.yml --experimental-upload-certs | tee kubeadm-init.log

2.2 安装成功则会有如下输出

[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.141.130]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kubernetes-master localhost] and IPs [192.168.141.130 127.0.0.1 ::1]
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kubernetes-master localhost] and IPs [192.168.141.130 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 20.003326 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in ConfigMap "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
2cd5b86c4905c54d68cc7dfecc2bf87195e9d5d90b4fff9832d9b22fc5e73f96
[mark-control-plane] Marking the node kubernetes-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kubernetes-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

# 后面子节点加入需要如下命令
kubeadm join 192.168.141.130:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:cab7c86212535adde6b8d1c7415e81847715cfc8629bb1d270b601744d662515

注意：如果安装 kubernetes 版本和下载的镜像版本不统一则会出现 timed out waiting for the condition 错误。中途失败或是想修改配置可以使用 kubeadm reset 命令重置配置，再做初始化操作即可。

2.3 配置 kubectl

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

# 非 ROOT 用户执行
chown $(id -u):$(id -g) $HOME/.kube/config

2.4 验证是否成功

kubectl get node

# 能够打印出节点信息即表示成功
NAME                STATUS     ROLES    AGE     VERSION
kubernetes-master   NotReady   master   8m40s   v1.14.1

至此 主节点初始化成功

2.5 kubeadm init 的执行过程

• init：指定版本进行初始化操作
• preflight：初始化前的检查和下载所需要的 Docker 镜像文件
• kubelet-start：生成 kubelet 的配置文件 var/lib/kubelet/config.yaml，没有这个文件 kubelet 无法启动，所以初始化之前的 kubelet 实际上启动不会成功
• certificates：生成 Kubernetes 使用的证书，存放在 /etc/kubernetes/pki 目录中
• kubeconfig：生成 KubeConfig 文件，存放在 /etc/kubernetes 目录中，组件之间通信需要使用对应文件
• control-plane：使用 /etc/kubernetes/manifest 目录下的 YAML 文件，安装 Master 组件
• etcd：使用 /etc/kubernetes/manifest/etcd.yaml 安装 Etcd 服务
• wait-control-plane：等待 control-plan 部署的 Master 组件启动
• apiclient：检查 Master 组件服务状态。
• uploadconfig：更新配置
• kubelet：使用 configMap 配置 kubelet
• patchnode：更新 CNI 信息到 Node 上，通过注释的方式记录
• mark-control-plane：为当前节点打标签，打了角色 Master，和不可调度标签，这样默认就不会使用 Master 节点来运行 Pod
• bootstrap-token：生成 token 记录下来，后边使用 kubeadm join 往集群中添加节点时会用到
• addons：安装附加组件 CoreDNS 和 kube-proxy

三、配置node节点 和集群网络

1 配置node节点

1.1 概述

将 slave 节点加入到集群中很简单，只需要在 slave 服务器上安装 kubeadm，kubectl，kubelet 三个工具，然后使用 kubeadm join 命令加入即可。准备工作如下：

• 修改主机名

  • 查看主机名

    # 查看当前主机名
    hostnamectl
    # 显示如下内容
       Static hostname: ubuntu
             Icon name: computer-vm
               Chassis: vm
            Machine ID: 33011e0a95094672b99a198eff07f652
               Boot ID: dc856039f0d24164a9f8a50c506be96d
        Virtualization: vmware
      Operating System: Ubuntu 18.04.2 LTS
                Kernel: Linux 4.15.0-48-generic
          Architecture: x86-64
    

  • 修改主机名

    # 使用 hostnamectl 命令修改，其中 kubernetes-master 为新的主机名
    hostnamectl set-hostname kubernetes-slave1
    

  • 修改cloud.cfg

    # 如果有该文件
    vi /etc/cloud/cloud.cfg
    
    # 该配置默认为 false，修改为 true 即可
    preserve_hostname: true
    

  • 验证是否已修改了主机名

    root@kubernetes-master:~# hostnamectl
       Static hostname: kubernetes-master
             Icon name: computer-vm
               Chassis: vm
            Machine ID: 33011e0a95094672b99a198eff07f652
               Boot ID: 8c0fd75d08c644abaad3df565e6e4cbd
        Virtualization: vmware
      Operating System: Ubuntu 18.04.2 LTS
                Kernel: Linux 4.15.0-48-generic
          Architecture: x86-64
    

• 配置软件源

  • 安装系统工具

    apt-get update && apt-get install -y apt-transport-https
    

  • 安装GPG证书

    curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
    

  • 写入软件源

    cat << EOF >/etc/apt/sources.list.d/kubernetes.list
    	> deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
    	> EOF
    

• 安装三个工具

  • kubeadm，kubelet，kubectl

    apt-get update  
    apt-get install -y kubelet kubeadm kubectl
    

  • 设置 kubelet 自启动，并启动 kubelet

    systemctl enable kubelet && systemctl start kubelet
    

1.2 将 slave 加入到集群

kubeadm join 192.168.141.130:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:cab7c86212535adde6b8d1c7415e81847715cfc8629bb1d270b601744d662515

1.3 安装成功将看到如下信息

[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.14" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster

相关参数说明

• token
  • 可以通过安装 master 时的日志查看 token 信息
  • 可以通过 kubeadm token list 命令打印出 token 信息
  • 如果 token 过期，可以使用 kubeadm token create 命令创建新的 token
• discovery-token-ca-cert-hash
  • 可以通过安装 master 时的日志查看 sha256 信息
  • 可以通过以下命令查看 sha256 信息

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //

1.4 验证是否成功

• 查看命令

kubectl get nodes

• 查看结果: 可以看到 slave 成功加入 master

NAME                STATUS     ROLES    AGE   VERSION
kubernetes-master   NotReady   master   9h    v1.14.1
kubernetes-slave1   NotReady   <none>   22s   v1.14.1

1.5 查看 pod 状态

• 查看命令

kubectl get pod -n kube-system -o wide

• 查看结果

NAME                                        READY   STATUS    RESTARTS   AGE   IP                NODE                NOMINATED NODE   READINESS GATES
coredns-8686dcc4fd-gwrmb                    0/1     Pending   0          9h    <none>            <none>              <none>           <none>
coredns-8686dcc4fd-j6gfk                    0/1     Pending   0          9h    <none>            <none>              <none>           <none>
etcd-kubernetes-master                      1/1     Running   1          9h    192.168.141.130   kubernetes-master   <none>           <none>
kube-apiserver-kubernetes-master            1/1     Running   1          9h    192.168.141.130   kubernetes-master   <none>           <none>
kube-controller-manager-kubernetes-master   1/1     Running   1          9h    192.168.141.130   kubernetes-master   <none>           <none>
kube-proxy-496dr                            1/1     Running   0          17m   192.168.141.131   kubernetes-slave1   <none>           <none>
kube-proxy-rsnb6                            1/1     Running   1          9h    192.168.141.130   kubernetes-master   <none>           <none>
kube-scheduler-kubernetes-master            1/1     Running   1          9h    192.168.141.130   kubernetes-master   <none>           <none>

由此可以看出 coredns 尚未运行，此时我们还需要安装网络插件。

2 配置kubernates集群网络

2.1 网络简介

容器网络是容器选择连接到其他容器、主机和外部网络的机制。容器的 runtime 提供了各种网络模式，每种模式都会产生不同的体验。例如，Docker 默认情况下可以为容器配置以下网络：

• none： 将容器添加到一个容器专门的网络堆栈中，没有对外连接。
• host： 将容器添加到主机的网络堆栈中，没有隔离。
• default bridge： 默认网络模式。每个容器可以通过 IP 地址相互连接。
• 自定义网桥： 用户定义的网桥，具有更多的灵活性、隔离性和其他便利功能。

2.2 什么是 CNI

CNI(Container Network Interface) 是一个标准的，通用的接口。在容器平台，Docker，Kubernetes，Mesos 容器网络解决方案 flannel，calico，weave。只要提供一个标准的接口，就能为同样满足该协议的所有容器平台提供网络功能，而 CNI 正是这样的一个标准接口协议。

2.3 Kubernetes 中的 CNI 插件

CNI 的初衷是创建一个框架，用于在配置或销毁容器时动态配置适当的网络配置和资源。插件负责为接口配置和管理 IP 地址，并且通常提供与 IP 管理、每个容器的 IP 分配、以及多主机连接相关的功能。容器运行时会调用网络插件，从而在容器启动时分配 IP 地址并配置网络，并在删除容器时再次调用它以清理这些资源。

运行时或协调器决定了容器应该加入哪个网络以及它需要调用哪个插件。然后，插件会将接口添加到容器网络命名空间中，作为一个 veth 对的一侧。接着，它会在主机上进行更改，包括将 veth 的其他部分连接到网桥。再之后，它会通过调用单独的 IPAM（IP地址管理）插件来分配 IP 地址并设置路由。

在 Kubernetes 中，kubelet 可以在适当的时间调用它找到的插件，为通过 kubelet 启动的 pod进行自动的网络配置。

Kubernetes 中可选的 CNI 插件如下：

• Flannel
• Calico
• Canal
• Weave

2.4 什么是 Calico

Calico 为容器和虚拟机提供了安全的网络连接解决方案，并经过了大规模生产验证（在公有云和跨数千个集群节点中），可与 Kubernetes，OpenShift，Docker，Mesos，DC / OS 和 OpenStack 集成。

Calico 还提供网络安全规则的动态实施。使用 Calico 的简单策略语言，您可以实现对容器，虚拟机工作负载和裸机主机端点之间通信的细粒度控制

2.5 安装网络插件 Calico

注意：截止到文章发表日期，Calico 官方版本为 3.7 +

参考官方文档安装：https://docs.projectcalico.org/v3.7/getting-started/kubernetes/

• 安装集群 网络插件 Calico

# 安装网络
kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
# 观察网络插件是否启动成功
watch kubectl get pods --all-namespaces

• 查看网络插件 Calico 是否跑起来

Every 2.0s: kubectl get pods --all-namespaces                                                                                                    kubernetes-master: Fri May 10 18:16:51 2019

NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-8646dd497f-g2lln    1/1     Running   0          50m
kube-system   calico-node-8jrtp                           1/1     Running   0          50m
kube-system   coredns-8686dcc4fd-mhwfn                    1/1     Running   0          51m
kube-system   coredns-8686dcc4fd-xsxwk                    1/1     Running   0          51m
kube-system   etcd-kubernetes-master                      1/1     Running   0          50m
kube-system   kube-apiserver-kubernetes-master            1/1     Running   0          51m
kube-system   kube-controller-manager-kubernetes-master   1/1     Running   0          51m
kube-system   kube-proxy-p8mdw                            1/1     Running   0          51m
kube-system   kube-scheduler-kubernetes-master            1/1     Running   0          51m

至此基本网络环境已安装完毕