kubernetes 集群部署

kubernetes 集群部署

环境
JiaoJiao_Centos7-1(152.112) 192.168.152.112
JiaoJiao_Centos7-2(152.113) 192.168.152.113
JiaoJiao_Centos7-3(152.114) 192.168.152.114
已开通 4C+8G+80G

集群规划

部署方式

环境准备：基于主机名称通信，时间同步，关闭firewall和iptables.service

方式一：yum ,rpm 安装。复杂。

1. etcd cluster ：安装在master节点上2. flannel：安装在所有节点上
2. 安装配置master
kube-apiserver,kube-scheduler,kube-controller-manager
3. 安装配置node
kube-proxy
kubelet

方式二：使用kubeadm工具安装。简单。

1. master和node 都用yum 安装kubelet，kubeadm，docker
2. master 上初始化：kubeadm init
3. master 上启动一个flannel的pod
4. node上加入集群:kubeadm join

部署步骤

采取方式二：使用kubeadm工具安装步骤

准备环境

修改主机名（3台机器都需要修改）

# hostnamectl set-hostname master

配置hosts和dns解析（3台机器都需要配置）

[root@node02 ~]# cat /etc/hosts
127.0.0.1 node02 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 node02 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.152.112 master
192.168.152.113 node01
192.168.152.114 node02

默认的dns服务器没法解析到opsx.alibaba.com和mirrors.aliyun.com，使用nameserver 192.168.?.?（具体根据你所在公司环境设置）这个可以。

[root@master ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.?.?

关闭防火墙和iptables，检查iptables配置值是否为1（3台机器都需要）

# systemctl stop firewalld.service
# systemctl stop iptables.service 
# systemctl disable firewalld.service
[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables 
1
[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-iptables 
1

配置docker yum源

#yum update
#yum install -y yum-utils device-mapper-persistent-data lvm2 wget
#cd /etc/yum.repos.d
#wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

配置kubernetes yum 源

#wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
# rpm --import rpm-package-key.gpg
# rpm --import yum-key.gpg
[root@master yum.repos.d]# cat kubernetes.repo 
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1
# yum repolist

master和node 安装kubelet，kubeadm，docker

# yum install docker-ce kubelet kubeadm

master 上安装kubectl ，node节点可以选择安装或者不安装

kubectl 是API server 客户端程序，通过连接master端的API server来实现k8s对象资源的增删改查等基本操作。

#yum install kubectl

docker的配置

配置私有仓库和镜像加速地址

[root@master docker]# cat /etc/docker/daemon.json 
{"registry-mirrors": ["http://295c6a59.m.daocloud.io"],
"insecure-registries":["自己搭建的私有仓库地址"]}

设置代理

[root@master docker]# cat /usr/lib/systemd/system/docker.service |grep -v "^#"
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
#设置代理地址，和不让代理的地址
Environment="HTTPS_PROXY=http://www.ik8s.io:10080" "NO_PROXY=localhost,127.0.0.1,192.168.152.0/24"
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target

启动docker

# systemctl daemon-reload 
# systemctl start docker 
[root@master docker]# docker info 
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64GiB
Name: master
ID: KEBQ:4XXU:LTA7:J5QH:HK4F:JS26:RKI4:YFPJ:RY45:Q647:SJI6:VZDQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTPS Proxy: http://www.ik8s.io:10080 #代理地址
No Proxy: localhost,127.0.0.1,192.168.152.0/24
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
XXXXX #本公司私有库地址
127.0.0.0/8
Registry Mirrors:
http://295c6a59.m.daocloud.io/ #镜像加速地址
Live Restore Enabled: false
Product License: Community Engine 
# systemctl enable docker

master 上初始化：kubeadm init

修改配置文件，忽略swap报错

[root@master docker]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

初始化

#kubeadm init [command] --help 可以先查看下帮助
# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap

--kubernetes-version=v1.14 # k8s的版本
版本通过：https://github.com/kubernetes/kubernetes/releases 来确定你需要的版本。
--pod-network-cidr=10.244.0.0/16 #pod网络。flannel 的默认是10.244.0.0/16
--service-cidr=10.96.0.0/12 # service 网络，也是用了默认值
--ignore-preflight-errors=Swap #忽略Swap报错

[root@master ~]# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1

报错了。拉不到镜像。
解决办法：docker.io仓库对google的容器做了镜像，可以通过下列命令下拉取相关镜像

#docker pull mirrorgooglecontainers/kube-apiserver:v1.14.2
#docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.2
#docker pull mirrorgooglecontainers/kube-scheduler:v1.14.2
#docker pull mirrorgooglecontainers/kube-proxy:v1.14.2
#docker pull mirrorgooglecontainers/pause:3.1
#docker pull mirrorgooglecontainers/etcd:3.3.10
#docker pull coredns/coredns:1.3.1

通过docker tag命令来修改镜像的标签：

# docker tag mirrorgooglecontainers/kube-apiserver:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2
# docker tag mirrorgooglecontainers/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2
# docker tag mirrorgooglecontainers/kube-scheduler:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2 
# docker tag mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2 
# docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1 
# docker tag mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
# docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1 

初始化过程中有提示，请按照提示进行操作并且将下面内容保存下来。

kubeadm join 192.168.152.112:6443 --token qlu35g.rfxl6k8mvry8nxfp \
> --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d

部署网络：master 上启动一个flannel的pod

下载镜像。由于不能直接下载官方镜像，自己百度找了个地址：

#docker pull jmgao1983/flannel:v0.11.0-amd64
#docker tag jmgao1983/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
#docker rmi jmgao1983/flannel:v0.11.0-amd64

启动：

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

检查下：

[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-f7db6 1/1 Running 0 53m
coredns-fb8b8dccf-sd8hk 1/1 Running 0 53m
etcd-master 1/1 Running 0 52m
kube-apiserver-master 1/1 Running 0 52m
kube-controller-manager-master 1/1 Running 0 52m
kube-flannel-ds-amd64-zq5zr 1/1 Running 0 19m
kube-proxy-8mtbc 1/1 Running 0 53m
kube-scheduler-master 1/1 Running 0 52m

到此，master 就安装好了! 保存镜像

docker save -o master.tar k8s.gcr.io/kube-proxy:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2 quay.io/coreos/flannel:v0.11.0-amd64 k8s.gcr.io/coredns:1.3.1 k8s.gcr.io/etcd:3.3.10 k8s.gcr.io/pause:3.1

node节点加入集群

将master上保存的镜像同步到节点上

[root@master ~]#scp master.tar node01:/root/
[root@master ~]#scp master.tar node02:/root/

将镜像导入本地

[root@node01 ~]# docker load< master.tar

修改配置，忽略报错：

[root@node01 docker]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

执行加入集群命令

[root@node01 ~]# kubeadm join 192.168.152.112:6443 --token qlu35g.rfxl6k8mvry8nxfp --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d --ignore-preflight-errors=Swap

在master上查看节点情况

[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 107m v1.14.2
node01 Ready <none> 23m v1.14.2
node02 Ready <none> 23m v1.14.2