How to install Kubernetes happily in China

Installing K8S in China has always been a headache for everyone. The key is that you don't know what tools you need to download. It hurts. And most domestic tutorials are either too old or too frustrating. Today, I will tell you how to install K8S happily in China.

Installation Environment

I use kubeadm for installation, and the process basically comes from the official tutorial.

Category	model
platform	Alibaba Cloud VPC
system	Centos 7.3
Docker version	1.12.6
K8S version	1.6.*

Although it is written in Alibaba Cloud VPC, virtual machines are also supported. Has little effect.

 

node	effect	quantity	Recommended configuration
Master	K8S master node (etcd, API, controller...)	1	1 core 2G
Node	application node	2	2 cores 4G

If you just install and play locally, you can configure it to 1 core and 1G.

In the next process, if there is a situation where the host is over the wall, you can follow the process. If not, you can just look at the downloaded result.

 

Installation process

The installation process basically includes downloading the software, downloading the image, configuring the host, starting the Master node, configuring the network, and starting the Node node.

Download software

People who have the ability to bypass the wall in this step can follow the steps directly. If not, they can use the 1.6.2 software package I packaged.

First, configure the K8S source on your own over-the-wall host.

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

Once configured, download the package

yum install -y -downloadonly kubelet kubeadm kubectl kubernetes-cni

 Package all downloaded RPMs and send them back to the local. This completes the download of the K8S software package.

Downloaded RPMs:

https://pan.baidu.com/s/1clIpjC cp6h

Download mirror

To download the mirror, you can directly use the script I provided, as long as you can bypass the wall, you know. I have already downloaded one myself. Available to everyone.

#!/usr/bin/env bash

images=(
    kube-proxy-amd64:v1.6.2
    kube-controller-manager-amd64:v1.6.2
    kube-apiserver-amd64:v1.6.2
    kube-scheduler-amd64:v1.6.2
    kubernetes-dashboard-amd64:v1.6.0
    k8s-dns-sidecar-amd64:1.14.1
    k8s-dns-kube-dns-amd64:1.14.1
    k8s-dns-dnsmasq-nanny-amd64:1.14.1
    etcd-amd64:3.0.17
    pause-amd64:3.0
)

for imageName in ${images[@]} ; do
    docker pull gcr.io/google_containers/$imageName
    docker tag gcr.io/google_containers/$imageName registry.cn-beijing.aliyuncs.com/bbt_k8s/$imageName
    docker push registry.cn-beijing.aliyuncs.com/bbt_k8s/$imageName
done

quay.io/coreos/flannel:v0.7.0-amd64
docker tag quay.io/coreos/flannel:v0.7.0-amd64 registry.cn-beijing.aliyuncs.com/bbt_k8s/flannel:v0.7.0-amd64
docker push registry.cn-beijing.aliyuncs.com/bbt_k8s/flannel:v0.7.0-amd64

Let me explain about this script. This script is to download commonly used images, and then send them back to the domestic source. You can change registry.cn-beijing.aliyuncs.com/bbt_k8s to your own address. You must use docker login to log in first, otherwise permissions may appear. Authentication error. It is recommended to use the services of Alibaba Cloud and NetEase. If you use your own address, please pay attention to the following configuration, modify it according to your own situation, it is no longer cumbersome.

 

Mirror version number description:

software	Version	illustrate	Remark
kube-proxy-amd64 kube-controller-manager-amd64 kube-apiserver-amd64 kube-scheduler-amd64	v1.6.2	These images generally follow the version of K8S. For example, if I installed 1.6.2 of K8S, then the version number is v1.6.2
kubernetes-dashboard-amd64	v1.6.0	This is the K8S console (although it is not easy to use, but at least it is suitable for novices), it is generally easy to follow the large version of K8S. For example, I installed K8S 1.6.2, the large version is 1.6, so the version number is v1 .6.0
k8s-dns-sidecar-amd64 k8s-dns-kube-dns-amd64 k8s-dns-dnsmasq-nanny-amd64	1.14.1	This is a DNS service, which is generally not upgraded with K8S. For the specific version, please refer to https://kubernetes.io/docs/getting-started-guides/kubeadm/
etcd-amd64	3.0.17	This is the etcd service, which is generally not upgraded with K8S. For the specific version, please refer to https://kubernetes.io/docs/getting-started-guides/kubeadm/
pause-amd64	3.0	Generally do not follow K8S to upgrade, the specific version can refer to https://kubernetes.io/docs/getting-started-guides/kubeadm/	It has been version 3.0 for a long time.
flannel	v0.7.0-amd64	The network component, the flannel I use here, can of course use other. For specific version information, refer to the corresponding network components. For example, flannel is https://github.com/coreos/flannel/tree/master/Documentation

OK,这些镜像下载完成就 OK 了，没有翻墙工具的，就直接跳过吧。

 

主机配置

上面的内容下载好后，我们就可以安装了。

更新系统

没什么可以介绍的。

yum update -y

安装 Docker

K8S 的1.6.x 版本仅仅在 Docker 1.12上测试过，虽然最新版本 Docker 也可以运行，但是不推荐安装最新版本，免得遇到什么问题。

curl -sSL http://acs-public-mirror.oss-cn-hangzhou.aliyuncs.com/docker-engine/internet | sh /dev/stdin 1.12.6

安装完成后，禁用 Docker 的更新，禁用方式为，在/etc/yum.conf添加

exclude=docker-engine*

配置 Docker

主要是配置一些加速器，避免自己下载镜像速度太慢。

修改/etc/docker/daemon.json 添加如下内容：

{
  "registry-mirrors": ["https://自己的加速地址"]
}

之后就是启动Docker 的服务，

systemctl daemon-reload
systemctl enable docker
systemctl start docker

 

修改网络

主要是开启桥接相关支持，这个是 flannel 需要的配置，具体是否需要，看自己的网络组件选择的是什么。

修改/usr/lib/sysctl.d/00-system.conf,将net.bridge.bridge-nf-call-iptables改成1.之后修改当前内核状态

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

 

安装K8S包

上传RPM 包，到自己的服务器上，然后执行

yum install -y *.rpm

之后开启kubelet的开机启动

systemctl enable kubelet

然后配置 kubelet，修改/etc/systemd/system/kubelet.service.d/10-kubeadm.conf成如下文件

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_ALIYUN_ARGS=--pod-infra-container-image=registry-vpc.cn-beijing.aliyuncs.com/bbt_k8s/pause-amd64:3.0"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS $KUBELET_ALIYUN_ARGS

在这里主要修正2个问题，一个是将 POD 的基础容器修改为我们自己源里面的，另外一个是最新版本的 K8S 的资源管理和 Docker 默认的资源管理方式有冲突，这里把这块给删除了。具体可以参考https://github.com/kubernetes/release/issues/306

然后重新 reload 服务。

systemctl daemon-reload

这样就完成了主机环境的初始化，如果是使用虚拟机，拷贝3份就可以了。如果是实体机，3台都按照这个步骤来一遍就好了。然后为每一台主机根据类型设置好 HostName，K8S会把 HostName 当做主机标识。

启动 Master

配置完主机后，我们就可以启动我们的 Master 节点了，通常 Master 节点推荐2-3个，本地测试我们就简单一些，一个节点就可以了。

export KUBE_REPO_PREFIX="registry-vpc.cn-beijing.aliyuncs.com/bbt_k8s"
export KUBE_ETCD_IMAGE="registry-vpc.cn-beijing.aliyuncs.com/bbt_k8s/etcd-amd64:3.0.17"
kubeadm init --kubernetes-version=v1.6.2 --pod-network-cidr=10.96.0.0/12

前面2个环境变量配置，是让 kubeadm 初始化的时候，使用我们的镜像源下载镜像。

最后 kubeadm init 是初始化 Master 节点。其中需要配置的参数我说明一下。

参数	意义	备注
--kubernetes-version	K8S 的版本号，根据自己下载的镜像和 RPM 版本选择。	我这里使用的1.6.2，所以版本为v1.6.2.
--pod-network-cidr	POD 的网络，只要不和主机网络冲突就可以，我这里使用的是10.96.0.0/12	这个和上面/etc/systemd/system/kubelet.service.d/10-kubeadm.conf里面声明的KUBELET_DNS_ARGS挂钩，请一同修改。

执行完毕后，稍等一阵，就完成了。

kubeadm init --kubernetes-version=v1.6.2 --pod-network-cidr=10.96.0.0/12
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.2
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [node0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.61.41]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 14.583864 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 6.008990 seconds
[token] Using token: e7986d.e440de5882342711
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token 1111.1111111111111 *.*.*.*:6443

安装完成后，有一个内容非常重要，查看安装日志，拷贝类似于下面的语句，这条语句用来初始化之后的节点。

kubeadm join --token 11111.11111111111111 *.*.*.*:6443

 

接下来我们就可以去看看我们 K8S 的状态。我这使用的是 Mac。 Linux 和 Windows 的同学自行处理。

安装 kubectl

brew install kubectl

然后拷贝 Master 节点上的/etc/kubernetes/admin.conf文件到本机的~/.kube/config

之后执行kebectl get node。我这里已经安装完毕了，所以有全部信息，只要能看到节点，就算是成功了。

安装网络组件

接下来我们安装网络组件，我这里使用的是flannel。创建2个文件

kube-flannel-rbac.yml

# Create the clusterrole and clusterrolebinding:
# $ kubectl create -f kube-flannel-rbac.yml
# Create the pod using the same namespace used by the flannel serviceaccount:
# $ kubectl create --namespace kube-system -f kube-flannel.yml
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system

kube-flannel-ds.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.96.0.0/12",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      hostNetwork: true
      nodeSelector:
        beta.kubernetes.io/arch: amd64
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      containers:
      - name: kube-flannel
        image: registry.cn-beijing.aliyuncs.com/bbt_k8s/flannel:v0.7.0-amd64
        command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
        securityContext:
          privileged: true
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: /run
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      - name: install-cni
        image: registry.cn-beijing.aliyuncs.com/bbt_k8s/flannel:v0.7.0-amd64
        command: [ "/bin/sh", "-c", "set -e -x; cp -f /etc/kube-flannel/cni-conf.json /etc/cni/net.d/10-flannel.conf; while true; do sleep 3600; done" ]
        volumeMounts:
        - name: cni
          mountPath: /etc/cni/net.d
        - name: flannel-cfg
          mountPath: /etc/kube-flannel/
      volumes:
        - name: run
          hostPath:
            path: /run
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        - name: flannel-cfg
          configMap:
            name: kube-flannel-cfg

然后使用使用命令进行配置。

kubectl create -f kube-flannel-rbac.yml
kubectl create -f kube-flannel-ds.yaml

 

启动 Node

分别在2个Node 节点,执行下面的命令。

export KUBE_REPO_PREFIX="registry-vpc.cn-beijing.aliyuncs.com/bbt_k8s"
export KUBE_ETCD_IMAGE="registry-vpc.cn-beijing.aliyuncs.com/bbt_k8s/etcd-amd64:3.0.17"
kubeadm join --token 1111.111111111111 *.*.*.*:6443

其中 kubeadm join 请参考启动 Master 节点中的内容。

 

其它

理论上我们安装到这里，K8S 就已经可以使用了。接下来主要是K8S的 Dashboard 的安装，仅供参考，不一定要安装。

安装 DashBoard 工具

创建文件kubernetes-dashboard.yaml

# Copyright 2015 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.6 (RBAC enabled).
#
# Example usage: kubectl create -f <this_file>

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kubernetes-dashboard
  template:
    metadata:
      labels:
        app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: registry.cn-beijing.aliyuncs.com/bbt_k8s/kubernetes-dashboard-amd64:v1.6.0
        imagePullPolicy: Always
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
        livenessProbe:
          httpGet:
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 9090
  selector:
    app: kubernetes-dashboard

 

创建文件dashboard-rbac.yaml

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: dashboard-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin 
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system

之后执行

kubectl create -f dashboard-rbac.yml
kubectl create -f kubernetes-dashboard.yaml

 

之后用下面的命令，获取到对应端口号。主要是看 NodePort:        <unset>    31867/TCP这样的内容。然后使用http://NodeIp:NodePort访问就可以了，其中 NodeIp 为 Master 或者 Node 的 IP，NodePort为NodePort的端口。

kubectl describe --namespace kube-system service kubernetes-dashboard

 

到此，这份安装教程就到这里结束了。最后奉上一份安装后的截图。