Kubernetes Series kubeadm deploy a multi-node cluster master

Version Information

linux   Centos7
kubernetes v1.14.5
docker v18.06.1-ce

Node information

VIP:47.110.19.11    阿里云负载均衡

Preparation Before Installation

开始部署前确保所有节点网络正常，能访问公网。主要操作都在VPC-OPEN-MASTER001节点进行，设置VPC-OPEN-MASTER001可以免密码登陆其他节点。所有操作都使用root用户身份进行。

Server Description

我们这里使用的是五台centos-7.6的虚拟机，具体信息如下表：
系统类型    IP地址            节点角色    CPU Memory  Hostname
centos-7.6  192.168.3.42    master      >=2 >=4G    master01
centos-7.6  192.168.3.43    master      >=2 >=4G    master02
centos-7.6  192.168.3.44    master      >=2 >=4G    master03
centos-7.6  192.168.3.45    worker      >=2 >=4G    node01
centos-7.6  192.168.3.46    worker      >=2 >=4G    node02

First, prepare the environment

1, set the hostname

# 查看主机名
$ hostname
# 修改主机名
$ hostnamectl set-hostname huoban-k8s-master01
# 配置host，使所有节点之间可以通过hostname互相访问

>2、配置hosts解析

vim /etc/hosts

Huoban-K8S-master01 192.168.3.42 master01
192.168.3.43 huoban-K8S-master02 master02
192.168.3.44 huoban-K8S-master03 master03
192.168.3.45 huoban-K8S-amdha01 amdha01
192.168.3.46 huoban-K8S-amdha02 amdha02

>3、安装依赖包

Yum update

$ yum update

Installation dependencies

$ yum install -y conntrack ipvsadm ipset jq sysstat curl iptables libseccomp

>4、关闭防火墙、swap，重置iptables

Turn off the firewall

$ systemctl stop firewalld && systemctl disable firewalld

Reset iptables

$ iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

Close swap

$ swapoff -a
$ sed -i '/swap/s/^(.*)$/#\1/g' /etc/fstab

Close selinux

$ setenforce 0

Close dnsmasq (Doing so may result docker container can not resolve domain)

$ service dnsmasq stop && systemctl disable dnsmasq

>5、系统参数设置

Making Profile

$ cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
EOF

Take effect file

$ sysctl -p /etc/sysctl.d/kubernetes.conf

二、安装docker

Ali cloud images using the warehouse

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo

Installation docker

View version that can be installed
yum List Docker-ce --showduplicates | the Sort -r
yum -y install Docker-ce-3-18.06.1.ce

Set docker startup parameters (optional)

- graph: Set docker data directory: Select the larger partition (here I do not need to configure the root directory, defaults to / var / lib / docker)

- exec-opts: Set cgroup driver (default is cgroupfs, does not recommend setting systemd)

- registry-mirrors arranged mirror acceleration docker

cat > /etc/docker/daemon.json <<EOF
{
"graph": "/docker/data/path",
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"registry-mirrors":["https://k9e55i4n.mirror.aliyuncs.com"]
}
EOF

Start docker service and add startup entries

systemctl start docker && systemctl enable docker

三、安装 kubeadm, kubelet 和 kubectl

>1、配置yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

>2、安装kubelet,kubeadm,kubectl

Find the version number installed

yum list kubeadm --showduplicates | sort -r

# Install the specified version
yum install -y kubelet-1.14.5 kubeadm- 1.14.5 kubectl-1.14.5

>3、查看安装情况

systemctl cat kubelet

Kubelet can be seen to set a system service, and generates a 10-kubeadm.conf kubelet.service two files

/etc/systemd/system/kubelet.service

[Unit]
Description = kubelet The Kubernetes Node Agent
Documentation = http://kubernetes.io/docs/

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

Note: This dropin only works with kubeadm and kubelet v1.11+

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

EnvironmentFile = / var / lib / kubelet / kubeadm flags.env

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

EnvironmentGoogle nmentFile ro = - / etc / sysconfig / omelet
ExecStart =
ExecStart = / usr / bin / omelet KUBELET_KUBECONFIG_ARGS $ $ $ KUBELET_CONFIG_ARGS KUBELET_KUBEADM_ARGS $ KUBELET_EXTRA_ARGS

四、配置系统相关参数

# The following operations on all nodes operate
! / Bin / bash #

Open forward

Docker adjust the default firewall rules from the start version 1.13

Disable the iptables filter table FOWARD chain

This will cause Kubernetes cluster of Pod can not communicate across Node

iptables -P FORWARD ACCEPT

Ipvs related kernel module is loaded

If you reboot, you need to reload

modprobe ip_vs
modprobe ip_vs_rr
modprobe ip_vs_wrr
modprobe ip_vs_sh
modprobe nf_conntrack_ipv4
lsmod | grep ip_vs

五、配置阿里云负载均衡及修改证书

Download Source Package #
cd / usr / local / src /
git clone https://github.com/kubernetes/kubernetes.git
git Checkout -b Kubernetes-1.14.5 Origin / 1.14-Release

#docker pull modified image, the corresponding version has 1.11.5,1.12.3,1.13.0,1.13.2,1.13.4
Docker pull icyboy / k8s_build: v1.14.1

# k8s-1.14 or more modified two files expiration date, and modification date field to find NotAfter period
/usr/local/src/kubernetes/staging/src/k8s.io/client-go/util/cert/cert.go
NotAfter: now .add (duration365d 100) .UTC (),
/usr/local/src/kubernetes/cmd/kubeadm/app/util/pkiutil/pki_helpers.go
NotAfter: Time.now () the Add (duration365d. 100) .UTC () , # 100 in change

# Compile
docker run --rm -v /usr/local/src/kubernetes:/go/src/k8s.io/kubernetes -it icyboy / k8s_build: v1.14.1 bash

Compile kubeadm, here mainly to compile kubeadm

make all WHAT=cmd/kubeadm GOFLAGS=-v

Compile kubelet

make all WHAT=cmd/kubelet GOFLAGS=-v

Compile kubectl

make all WHAT=cmd/kubectl GOFLAGS=-v

# Compiled product under / usr / local / src / kubernetes / _output / local / bin / linux / amd64 catalog
# kubeadm file copy out, replace the system kubeadm

# Replace with a new official kubeadm kubeadm
chmod + the X-kubeadm && \ cp -f kubeadm / usr / bin

六、部署第一个主节点
>1、配置kubelet

The following operations need to be performed on all nodes

Reload kubelet system configuration

systemctl daemon-reload

Set boot, not this start kubelet

systemctl enable kubelet

>2、根据配置文件初始化集群

Use kubeadm-config.yaml cluster configuration k8s1.14.5

cat init.sh

LOAD_BALANCER_DNS="47.110.19.11"
LOAD_BALANCER_PORT="6443"

Generate kubeadm profile

cat > kubeadm-master.config <<EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration

kubernetes version

kubernetesVersion: v1.14.5

Ali mirroring domestic use

imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServer:
certSANs:

• "$LOAD_BALANCER_DNS"
  controlPlaneEndpoint: "$LOAD_BALANCER_DNS:$LOAD_BALANCER_PORT"

networking:
podSubnet: 10.244.0.0/16
EOF

# Initialize k8s cluster
kubeadm init --config = kubeadm-master.config

>3、验证证书有效时间

cd / etc / kubernetes / PKI

for crt in $(find /etc/kubernetes/pki/ -name "*.crt"); do openssl x509 -in $crt -noout -dates; done

notBefore=Aug 20 07:43:46 2019 GMT
notAfter=Jul 27 07:43:46 2119 GMT
notBefore=Aug 20 07:43:45 2019 GMT
notAfter=Jul 27 07:43:45 2119 GMT
notBefore=Aug 20 07:43:46 2019 GMT
notAfter=Jul 27 07:43:47 2119 GMT
notBefore=Aug 20 07:43:46 2019 GMT
notAfter=Jul 27 07:43:47 2119 GMT
notBefore=Aug 20 07:43:46 2019 GMT
notAfter=Jul 27 07:43:47 2119 GMT
notBefore=Aug 20 07:43:46 2019 GMT
notAfter=Jul 27 07:43:46 2119 GMT
notBefore=Aug 20 07:43:45 2019 GMT
notAfter=Jul 27 07:43:46 2119 GMT
notBefore=Aug 20 07:43:45 2019 GMT
notAfter=Jul 27 07:43:45 2119 GMT
notBefore=Aug 20 07:43:45 2019 GMT
notAfter=Jul 27 07:43:45 2119 GMT
notBefore=Aug 20 07:43:45 2019 GMT
notAfter=Jul 27 07:43:45 2119 GMT

notBefore on behalf of the effective time, notAfter represents the expiration time.

>4、安装网络插件

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

>5、拷贝master证书到其他节点

cat scp.sh

USER=root
CONTROL_PLANE_IPS="192.168.3.43 192.168.3.44"
for host in ${CONTROL_PLANE_IPS}; do
scp /etc/kubernetes/pki/ca.crt "${USER}"@$host:
scp /etc/kubernetes/pki/ca.key "${USER}"@$host:
scp /etc/kubernetes/pki/sa.key "${USER}"@$host:
scp /etc/kubernetes/pki/sa.pub "${USER}"@$host:
scp /etc/kubernetes/pki/front-proxy-ca.crt "${USER}"@$host:
scp /etc/kubernetes/pki/front-proxy-ca.key "${USER}"@$host:
scp /etc/kubernetes/pki/etcd/ca.crt "${USER}"@$host:etcd-ca.crt
scp /etc/kubernetes/pki/etcd/ca.key "${USER}"@$host:etcd-ca.key
scp /etc/kubernetes/admin.conf "${USER}"@$host:
ssh ${USER}@${host} 'mkdir -p / etc / kubernetes / PKI / ETCs'
ssh $ @ $ USER} {} {Host '/${USER}/ca.crt mv / etc / kubernetes / PKI /'
ssh $ @ $ USER} {} {Host 'mv /${USER}/ca.key the / etc / kubernetes / PKI / '
ssh $ @ $ USER} {} {host' /${USER}/sa.pub mv / etc / kubernetes / PKI / '
ssh $ @ $ USER} {} {host "mv / {} $ USER /sa.key / etc / kubernetes / PKI / '
ssh $ @ $ USER} {} {host' /${USER}/front-proxy-ca.crt mv / etc / kubernetes / PKI / '
ssh $ @ $ USER} {} {host '/${USER}/front-proxy-ca.key mv / etc / kubernetes / PKI /'
ssh $ @ $ USER} {} {host "mv / {$ USER} / ETCs-ca.crt /etc/kubernetes/pki/etcd/ca.crt '
ssh $ @ $ USER} {} {host' /${USER}/etcd-ca.key mv / etc / kubernetes / PKI / ETCs / ca.key '
ssh $ @ $ USER} {} {host' mv /${USER}/admin.conf /etc/kubernetes/admin.conf '
done

七、其他节点上部署

#master

kubeadm join 47.110.19.11:6443 --token qlrq5y.1yhm3rz9r7ynfqf1 --discovery-token-ca-cert-hash sha256:62579157003c3537deb44b30f652c500e7fa6505b5ef6826d796ba1245283899 --experimental-control-plane

#node

kubeadm join 47.110.19.11:6443 --token qlrq5y.1yhm3rz9r7ynfqf1 --discovery-token-ca-cert-hash sha256:62579157003c3537deb44b30f652c500e7fa6505b5ef6826d796ba1245283899