基于Kubernetes构建和部署企业容器云

其他 2020-06-20 10:54:50 阅读次数: 0

目录

第一章：Kubernetes介绍和环境准备
一、什么是K8S
二、主要功能
三、服务介绍
3.1 master节点
3.2 node节点
四、实验环境准备
4.1 规划
4.2 网络设置
4.3 配置静态IP地址
4.4 关闭selinux、防火墙、networkmanage
4.5 设置主机名解析
4.6 配置epel源
4.7 配置免秘钥登录
第二章：Kubernetes集群初始化
一、安装Docker
二、准备部署目录
三、准备软件包
四、解压软件包
五、修改环境变量
六、手动制作CA证书
6.1 安装 CFSSL
6.2 初始化CFSSL
6.3 创建用来生成 CA 文件的 JSON 配置文件
6.4 创建用来生成 CA 证书签名请求（CSR）的 JSON 配置文件
6.5 生成CA证书（ca.pem）和密钥（ca-key.pem）
6.6 分发证书
七、ETCD集群部署
7.1 准备etcd软件包
7.2 创建 etcd 证书签名请求
7.4 生成 etcd 证书和私钥
7.4 证书移动到/opt/kubernetes/ssl目录下
7.5 设置ETCD配置文件
7.6 创建ETCD系统服务
7.7 重新加载系统服务
7.8 验证集群
第三章：Kubernetes的master节点部署
一、部署API server服务
1.1 准备软件包
1.2 创建生成CSR的 JSON 配置文件
1.3 生成 kubernetes 证书和私钥
1.4 创建 kube-apiserver 使用的客户端 token 文件
1.5 创建基础用户名/密码认证配置
1.6 部署Kubernetes API Server
1.7 启动APIserver
二、部署Controller Manager服务
三、部署Kubernetes Scheduler服务
四、部署kubectl 命令行工具
第四章：Kubernetes的node节点部署
4.1-kubelet部署
4.2-kube-proxy部署
第五章：Flannel网络部署
5.1 部署Flanne
5.2 创建一个K8S应用

第一章：Kubernetes介绍和环境准备

一、什么是K8S

Kubernetes是一个全新的基于容器技术的分布式架构领先方案，是一个开放的开发平台。不局限于任何一种语言，没有限定任何编程接口。是一个完备的分布式系统支撑平台。它构建在docker之上，提供应用部署、维护、扩展机制等功能，利用Kubernetes能方便地管理跨机器运行容器化的应用。

二、主要功能

  • 使用Docker对应用程序包装、实例化；
  • 以集群的方式运行、管理跨机器的容器；
  • 解决Docker跨机器容器之间的通讯问题；
  • Kubernetes的自我修复机制使得容器集群总是运行在用户期望的状态；

三、服务介绍

 　　Kubernetes中的：Node、Pod、Replication Controller、Service等都可以看作一种“资源对象”，几乎所有的资源对象都可以通过kubectl工具（API调用）执行增、删、改、查等操作并将其保存在etcd中持久化存储。

  • Master：集群控制管理节点，所有的命令都经由master处理。
  • Node：是kubernetes集群的工作负载节点。Master为其分配工作，当某个Node宕机时，Master会将其工作负载自动转移到其他节点。
  • Pod：是kubernetes最重要也是最基本的概念。每个Pod都会包含一个 “根容器”，还会包含一个或者多个紧密相连的业务容器。
  • ETCD：k8s中，所有数据的存储以及操作记录都在etcd中进行存储，一旦故障，可能导致整个集群的瘫痪或者数据丢失。
  • Label：是一个key=value的键值对，其中key与value由用户自己指定。可以附加到各种资源对象上，一个资源对象可以定义任意数量的Label。
  • RC：Replication Controller声明某个Pod的副本数在任意时刻都符合某个预期值。定义包含如下： 

   　　　　　　    （1）Pod期待的副本数（replicas）

   　　　　　　（2）用于筛选目标Pod的Label Selector

  　　　　　　 （3）当Pod副本数小于期望时，用于新的创建Pod的模板template 

   　　　　　　（4）通过改变RC里的Pod副本数量，可以实现Pod的扩容或缩容功能

   　　　　　　（5）通过改变RC里Pod模板中的镜像版本，可以实现Pod的滚动升级功能

3.1 master节点

  • API Server

　　集群控制管理节点，所有的命令都经由master处理，所有资源增删改查的唯一入口。

　　提供集群管理的REST API接口，包括认证授权、数据校验以及集群状态变更等;

　　只有API Server才直接操作etcd;

　　其他模块通过API Server查询或修改数据;

　　提供其他模块之间的数据交互和通信的枢纽;

  • Scheduler

　　负责分配调度Pod到集群内的node节点；
　　监听kube-apiserver，查询还未分配Node的Pod；
　　根据调度策略为这些Pod分配节点；

  • Controller Manager

　　所有其他群集级别的功能，目前由控制器Manager执行。资源对象的自动化控制中心；

　　由一系列的控制器组成，它通过apiserver监控整个集群的状态，并确保集群处于预期的工作状态;

  • ETCD

　　所有持久化的状态信息存储在ETCD中。

3.2 node节点

 　　默认情况下，Node节点可动态增加到kubernetes集群中，一旦Node被纳入集群管理范围，kubelet会定时向Master汇报自身的情况，以及之前有哪些Pod在运行等，这样Master可以获知每个Node的资源使用情况，并实现高效均衡的资源调度策略。如果Node没有按时上报信息，则会被Master判断为失联，Node状态会被标记为Not Ready，随后Master会触发工作负载转移流程。

  • Kubelet

　　管理Pods以及容器、镜像、 Volume等，实现对集群对节点的管理。

  • Kube-proxy

　　提供网络代理以及负载均衡，实现与Service通信。

  • Docker Engine

　　负责节点的容器的管理工作。

四、实验环境准备

4.1 规划

本次实验是基于2个node+1个master，每个节点都跑一个ETCD。

1. 操作系统 CentOS-7.x-x86_64。
2. 关闭 iptables 和 SELinux。
3. 所有节点的主机名和 IP 地址，使用/etc/hosts 做好主机名解析。
主机名 IP地址（NAT） CPU 内存 描述
k8s-master eth0 : 10.0.0.25 1VCPU 2G Kubernets Master节点/Etcd节点
k8s-node-1 eth0 : 10.0.0.26 1VCPU 2G Kubernets Node节点/ Etcd节点
k8s-node-1 eth0 : 10.0.0.27 1VCPU 2G Kubernets Node节点/ Etcd节点

4.2 网络设置

4.3 配置静态IP地址

复制代码

#将 UUID 和 MAC 地址已经其它配置删除掉，3个节点除了IP和主机名不同其他相同。
[root@k8s-master ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
TYPE=Ethernet
BOOTPROTO=static
NAME=eth0
DEVICE=eth0
ONBOOT=yes
IPADDR=10.0.0.25
NETMASK=255.255.255.0
GATEWAY=10.0.0.254
DNS=223.5.5.5

#重启网络服务
[root@k8s-master ~]# systemctl restart network

#设置 DNS 解析
[root@k8s-master ~]# vi /etc/resolv.conf
nameserver 223.5.5.5

复制代码

4.4 关闭selinux、防火墙、networkmanage

复制代码

setenforce 0
sed -i 's#SELINUX=enforcing#SELINUX=disabled#' /etc/selinux/config
systemctl disable firewalld.service
systemctl stop firewalld.service
systemctl stop NetworkManager
systemctl disable NetworkManager

复制代码

4.5 设置主机名解析

3个节点都做

复制代码

#3个节点都一样
[root@k8s-master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.25   k8s-master
10.0.0.26   k8s-node-1
10.0.0.27   k8s-node-2

复制代码

4.6 配置epel源

3个节点都做

复制代码

rpm -ivh http://mirrors.aliyun.com/epel/epel-release-latest-7.noarch.rpm
#下载常用命令
yum install -y net-tools vim lrzsz tree screen lsof tcpdump nc mtr nmap

#保证能上网
[root@k8s-master ~]# ping www.baidu.com -c3
PING www.a.shifen.com (61.135.169.121) 56(84) bytes of data.
64 bytes from 61.135.169.121: icmp_seq=1 ttl=128 time=5.41 ms
64 bytes from 61.135.169.121: icmp_seq=2 ttl=128 time=6.55 ms
64 bytes from 61.135.169.121: icmp_seq=3 ttl=128 time=8.97 ms

--- www.a.shifen.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2023ms
rtt min/avg/max/mdev = 5.418/6.981/8.974/1.486 ms

复制代码

4.7 配置免秘钥登录

只在master节点做

复制代码

[root@k8s-master ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
b1:a0:5b:02:57:0e:8f:1e:25:bf:46:1f:d1:f3:24:c4 root@k8s-master
The key's randomart image is:
+--[ RSA 2048]----+
|    o o .+.      |
|     X   .E .    |
|  . + * o  =     |
|   + + + +  .    |
|    + + S        |
|     =           |
|    .            |
|                 |
|                 |
+-----------------+
[root@k8s-master ~]# ssh-copy-id k8s-master
The authenticity of host 'k8s-master (10.0.0.25)' can't be established.
ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@k8s-master's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'k8s-master'"
and check to make sure that only the key(s) you wanted were added.

[root@k8s-master ~]# ssh-copy-id k8s-node-1
The authenticity of host 'k8s-node-1 (10.0.0.26)' can't be established.
ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@k8s-node-1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'k8s-node-1'"
and check to make sure that only the key(s) you wanted were added.

[root@k8s-master ~]# ssh-copy-id k8s-node-2
The authenticity of host 'k8s-node-2 (10.0.0.27)' can't be established.
ECDSA key fingerprint is 75:5c:83:a1:b4:cc:bf:28:71:a5:d5:d1:94:35:3c:9a.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@k8s-node-2's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'k8s-node-2'"
and check to make sure that only the key(s) you wanted were added.

复制代码

回到顶部

第二章：Kubernetes集群初始化

一、安装Docker

3个节点都做

复制代码

#第一步：使用国内Docker源
[root@k8s-master ~]# cd /etc/yum.repos.d/
[root@k8s-master yum.repos.d]# wget \https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#第二步：Docker安装：
[root@k8s-master ~]# yum install -y docker-ce
#第三步：启动后台进程：
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker

复制代码

二、准备部署目录

3个节点都做

复制代码

[root@k8s-master ~]# mkdir -p /opt/kubernetes/{cfg,bin,ssl,log}
[root@k8s-master ~]# tree -L 1 /opt/kubernetes/
/opt/kubernetes/
├── bin
├── cfg
├── log
└── ssl

复制代码

三、准备软件包

百度网盘下载地址：

链接: https://pan.baidu.com/s/1kUNV7t8SSF_yuP_WDaXj5g

密码: wexh

四、解压软件包

master节点做

复制代码

#将下载软件包上传到/usr/local/src/
[root@k8s-master ~]# ll /usr/local/src/
总用量 579812
-rw-r--r-- 1 root root 593725046 6月   2 13:27 k8s-v1.10.1-manual.zip
#解压
cd /usr/local/src/
unzip k8s-v1.10.1-manual.zip
cd /usr/local/src/k8s-v1.10.1-manual/k8s-v1.10.1
tar zxf kubernetes.tar.gz 
tar zxf kubernetes-server-linux-amd64.tar.gz 
tar zxf kubernetes-client-linux-amd64.tar.gz
tar zxf kubernetes-node-linux-amd64.tar.gz
mv ./* /usr/local/src/

复制代码

五、修改环境变量

3个节点都做

[root@k8s-master ~]# sed -i 's#PATH=$PATH:$HOME/bin#PATH=$PATH:$HOME/bin:/opt/kubernetes/bin#g' /root/.bash_profile 
[root@k8s-master ~]# source /root/.bash_profile

六、手动制作CA证书

6.1 安装 CFSSL

master节点做

复制代码

[root@k8s-master ~]# cd /usr/local/src/
[root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@k8s-master src]# wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@k8s-master src]# chmod +x cfssl*
[root@k8s-master src]# mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfo
[root@k8s-master src]# mv cfssljson_linux-amd64  /opt/kubernetes/bin/cfssljson
[root@k8s-master src]# mv cfssl_linux-amd64  /opt/kubernetes/bin/cfssl
#复制cfssl命令文件到k8s-node1和k8s-node2节点。如果实际中多个节点，就都需要同步复制。
[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.0.26: /opt/kubernetes/bin
[root@k8s-master src]# scp /opt/kubernetes/bin/cfssl* 10.0.0.27: /opt/kubernetes/bin

复制代码

6.2 初始化CFSSL

master节点做

[root@k8s-master src]# mkdir ssl && cd ssl
[root@k8s-master ssl]# cfssl print-defaults config > config.json
[root@k8s-master ssl]# cfssl print-defaults csr > csr.json

6.3 创建用来生成 CA 文件的 JSON 配置文件

master节点做

复制代码

[root@k8s-master ssl]# cd /usr/local/src/ssl
[root@k8s-master ssl]# vim ca-config.json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}

复制代码

6.4 创建用来生成 CA 证书签名请求（CSR）的 JSON 配置文件

master节点做

复制代码

[root@k8s-master ssl]# cd /usr/local/src/ssl
[root@k8s-master ssl]# vim ca-csr.json
{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

复制代码

6.5 生成CA证书（ca.pem）和密钥（ca-key.pem）

master节点做

复制代码

[root@k8s-master ssl]# cd /usr/local/src/ssl
[root@k8s-master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
[root@k8s-master ssl]#  ls -l ca*
-rw-r--r-- 1 root root  290 6月   2 15:58 ca-config.json
-rw-r--r-- 1 root root 1001 6月   2 15:58 ca.csr
-rw-r--r-- 1 root root  208 6月   2 15:58 ca-csr.json
-rw------- 1 root root 1675 6月   2 15:58 ca-key.pem
-rw-r--r-- 1 root root 1359 6月   2 15:58 ca.pem

复制代码

6.6 分发证书

master节点做

[root@k8s-master ssl]# cp ca.csr ca.pem ca-key.pem ca-config.json /opt/kubernetes/ssl
[root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.0.26:/opt/kubernetes/ssl 
[root@k8s-master ssl]# scp ca.csr ca.pem ca-key.pem ca-config.json 10.0.0.27:/opt/kubernetes/ssl

七、ETCD集群部署

7.1 准备etcd软件包

master节点做

复制代码

[root@k8s-master ~]# cd /usr/local/src/
[root@k8s-master src]# wget https://github.com/coreos/etcd/releases/download/v3.2.18/etcd-v3.2.18-linux-amd64.tar.gz
[root@k8s-master src]# tar zxf etcd-v3.2.18-linux-amd64.tar.gz
[root@k8s-master src]# cd etcd-v3.2.18-linux-amd64
[root@k8s-master etcd-v3.2.18-linux-amd64]# cp etcd etcdctl /opt/kubernetes/bin/ 
[root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.0.26:/opt/kubernetes/bin/
[root@k8s-master etcd-v3.2.18-linux-amd64]# scp etcd etcdctl 10.0.0.27:/opt/kubernetes/bin/

复制代码

7.2 创建 etcd 证书签名请求

master节点做

复制代码

[root@k8s-master ~]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# vim etcd-csr.json
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
"10.0.0.25",
"10.0.0.26",
"10.0.0.27"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

复制代码

7.4 生成 etcd 证书和私钥

master节点做

复制代码

[root@k8s-master ssl]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
  -ca-key=/opt/kubernetes/ssl/ca-key.pem \
  -config=/opt/kubernetes/ssl/ca-config.json \
  -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
#生成以下证书文件
[root@k8s-master ssl]#  ls -l etcd*
-rw-r--r-- 1 root root 1062 6月   3 10:48 etcd.csr
-rw-r--r-- 1 root root  275 6月   3 10:45 etcd-csr.json
-rw------- 1 root root 1675 6月   3 10:48 etcd-key.pem
-rw-r--r-- 1 root root 1436 6月   3 10:48 etcd.pem
[root@k8s-master ssl]#

复制代码

7.4 证书移动到/opt/kubernetes/ssl目录下

master节点做

[root@k8s-master ssl]# cp etcd*.pem /opt/kubernetes/ssl
[root@k8s-master ssl]# scp etcd*.pem 10.0.0.26:/opt/kubernetes/ssl 
[root@k8s-master ssl]# scp etcd*.pem 10.0.0.27:/opt/kubernetes/ssl

7.5 设置ETCD配置文件

master节点配置文件

复制代码

[root@k8s-master ~]#  vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-master"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://10.0.0.25:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.25:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.25:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.25:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

复制代码

node1节点配置文件

复制代码

[root@k8s-node-1 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://10.0.0.26:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.26:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.26:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.26:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

复制代码

node2节点配置文件

复制代码

[root@k8s-node-2 ~]# vim /opt/kubernetes/cfg/etcd.conf
#[member]
ETCD_NAME="etcd-node2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#ETCD_SNAPSHOT_COUNTER="10000"
#ETCD_HEARTBEAT_INTERVAL="100"
#ETCD_ELECTION_TIMEOUT="1000"
ETCD_LISTEN_PEER_URLS="https://10.0.0.27:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.27:2379,https://127.0.0.1:2379"
#ETCD_MAX_SNAPSHOTS="5"
#ETCD_MAX_WALS="5"
#ETCD_CORS=""
#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.27:2380"
# if you use different ETCD_NAME (e.g. test),
# set ETCD_INITIAL_CLUSTER value for this name, i.e. "test=http://..."
ETCD_INITIAL_CLUSTER="etcd-master=https://10.0.0.25:2380,etcd-node1=https://10.0.0.26:2380,etcd-node2=https://10.0.0.27:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.27:2379"
#[security]
CLIENT_CERT_AUTH="true"
ETCD_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"
PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_CA_FILE="/opt/kubernetes/ssl/ca.pem"
ETCD_PEER_CERT_FILE="/opt/kubernetes/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/opt/kubernetes/ssl/etcd-key.pem"

复制代码

7.6 创建ETCD系统服务

master节点做

复制代码

[root@k8s-master ~]# vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target

[Service]
Type=simple
WorkingDirectory=/var/lib/etcd
EnvironmentFile=-/opt/kubernetes/cfg/etcd.conf
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /opt/kubernetes/bin/etcd"
Type=notify

[Install]
WantedBy=multi-user.target
[root@k8s-master ~]# scp /etc/systemd/system/etcd.service 10.0.0.26:/etc/systemd/system/
[root@k8s-master ~]# scp /etc/systemd/system/etcd.service 10.0.0.27:/etc/systemd/system/

复制代码

7.7 重新加载系统服务

3个节点都做

systemctl daemon-reload
mkdir /var/lib/etcd
systemctl enable etcd
systemctl start etcd
systemctl status etcd

7.8 验证集群

master节点做

[root@k8s-master ~]# etcdctl --endpoints=https://10.0.0.25:2379 \
  --ca-file=/opt/kubernetes/ssl/ca.pem \
  --cert-file=/opt/kubernetes/ssl/etcd.pem \
  --key-file=/opt/kubernetes/ssl/etcd-key.pem cluster-health

如下图所示证明ETCD集群部署成功

回到顶部

第三章：Kubernetes的master节点部署

一、部署API server服务

1.1 准备软件包

master节点做

因为K8S是用二进制文件构写的所以不需要编译，只需要拷贝文件就行。

[root@k8s-master ~]# cd /usr/local/src/kubernetes
[root@k8s-master kubernetes]# cp server/bin/kube-apiserver /opt/kubernetes/bin/
[root@k8s-master kubernetes]# cp server/bin/kube-controller-manager /opt/kubernetes/bin/
[root@k8s-master kubernetes]# cp server/bin/kube-scheduler /opt/kubernetes/bin/

1.2 创建生成CSR的 JSON 配置文件

master节点做

复制代码

[root@k8s-master src]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# vim kubernetes-csr.json
{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "10.0.0.25",
    "10.1.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

复制代码

1.3 生成 kubernetes 证书和私钥

master节点做

复制代码

[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
[root@k8s-master ssl]# cp kubernetes*.pem /opt/kubernetes/ssl/
[root@k8s-master ssl]# scp kubernetes*.pem 10.0.0.26:/opt/kubernetes/ssl/
[root@k8s-master ssl]# scp kubernetes*.pem 10.0.0.27:/opt/kubernetes/ssl/

复制代码

1.4 创建 kube-apiserver 使用的客户端 token 文件

master节点做

[root@k8s-master ~]#  head -c 16 /dev/urandom | od -An -t x | tr -d ' '
841276913ac026513f75d267f0fc5212
[root@k8s-master ~]# vim /opt/kubernetes/ssl/bootstrap-token.csv
841276913ac026513f75d267f0fc5212,kubelet-bootstrap,10001,"system:kubelet-bootstrap"  #根据上面自己生成的填写

1.5 创建基础用户名/密码认证配置

master节点做

[root@k8s-master ~]#  vim /opt/kubernetes/ssl/basic-auth.csv
admin,admin,1
readonly,readonly,2

1.6 部署Kubernetes API Server

master节点做

复制代码

[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
ExecStart=/opt/kubernetes/bin/kube-apiserver \
  --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,NodeRestriction \
  --bind-address=10.0.0.25 \
  --insecure-bind-address=127.0.0.1 \
  --authorization-mode=Node,RBAC \
  --runtime-config=rbac.authorization.k8s.io/v1 \
  --kubelet-https=true \
  --anonymous-auth=false \
  --basic-auth-file=/opt/kubernetes/ssl/basic-auth.csv \
  --enable-bootstrap-token-auth \
  --token-auth-file=/opt/kubernetes/ssl/bootstrap-token.csv \
  --service-cluster-ip-range=10.1.0.0/16 \
  --service-node-port-range=20000-40000 \
  --tls-cert-file=/opt/kubernetes/ssl/kubernetes.pem \
  --tls-private-key-file=/opt/kubernetes/ssl/kubernetes-key.pem \
  --client-ca-file=/opt/kubernetes/ssl/ca.pem \
  --service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --etcd-cafile=/opt/kubernetes/ssl/ca.pem \
  --etcd-certfile=/opt/kubernetes/ssl/kubernetes.pem \
  --etcd-keyfile=/opt/kubernetes/ssl/kubernetes-key.pem \
  --etcd-servers=https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379 \
  --enable-swagger-ui=true \
  --allow-privileged=true \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/opt/kubernetes/log/api-audit.log \
  --event-ttl=1h \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

复制代码

1.7 启动APIserver

master节点做

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable kube-apiserver
[root@k8s-master ~]# systemctl start kube-apiserver
[root@k8s-master ~]# systemctl status kube-apiserver

二、部署Controller Manager服务

master节点做

复制代码

[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-controller-manager \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --allocate-node-cidrs=true \
  --service-cluster-ip-range=10.1.0.0/16 \
  --cluster-cidr=10.2.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
  --cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
  --root-ca-file=/opt/kubernetes/ssl/ca.pem \
  --leader-elect=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

###启动Controller Manager
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable kube-controller-manager
[root@k8s-master ~]# systemctl start kube-controller-manager
[root@k8s-master ~]# systemctl status kube-controller-manager

复制代码

三、部署Kubernetes Scheduler服务

master节点上做

复制代码

[root@k8s-master ~]# vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
ExecStart=/opt/kubernetes/bin/kube-scheduler \
  --address=127.0.0.1 \
  --master=http://127.0.0.1:8080 \
  --leader-elect=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

###启动scheduler服务
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable kube-scheduler
[root@k8s-master ~]# systemctl start kube-scheduler
[root@k8s-master ~]# systemctl status kube-scheduler

复制代码

四、部署kubectl 命令行工具

master节点做

复制代码

###准备二进制命令包
[root@k8s-master ~]# cd /usr/local/src/kubernetes/client/bin
[root@k8s-master bin]# cp kubectl /opt/kubernetes/bin/

###创建 admin 证书签名请求
[root@linux-node1 ~]# cd /usr/local/src/ssl/
[root@linux-node1 ssl]# vim admin-csr.json
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

###生成 admin 证书和私钥
[root@k8s-master ssl]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem    -ca-key=/opt/kubernetes/ssl/ca-key.pem    -config=/opt/kubernetes/ssl/ca-config.json    -profile=kubernetes admin-csr.json | cfssljson -bare admin
[root@k8s-master ssl]# mv admin*.pem /opt/kubernetes/ssl/

###设置集群参数
[root@k8s-master ssl]# kubectl config set-cluster kubernetes \
   --certificate-authority=/opt/kubernetes/ssl/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.25:6443

###设置客户端认证参数
[root@k8s-master ssl]# kubectl config set-credentials admin \
   --client-certificate=/opt/kubernetes/ssl/admin.pem \
   --embed-certs=true \
   --client-key=/opt/kubernetes/ssl/admin-key.pem

###设置上下文参数
[root@k8s-master ssl]# kubectl config set-context kubernetes \
   --cluster=kubernetes \
   --user=admin

###使用kubectl工具
[root@k8s-master ~]# ll /root/.kube/config 
[root@k8s-master ~]# kubectl get cs

复制代码

如下图所示证明master节点部署成功

回到顶部

第四章：Kubernetes的node节点部署

4.1-kubelet部署

master节点做

复制代码

###二进制包准备
[root@k8s-master ~]# cd /usr/local/src/kubernetes/server/bin/
[root@k8s-master bin]# cp kubelet kube-proxy /opt/kubernetes/bin/
[root@k8s-master bin]# scp kubelet kube-proxy 10.0.0.26:/opt/kubernetes/bin/
[root@k8s-master bin]# scp kubelet kube-proxy 10.0.0.27:/opt/kubernetes/bin/

###创建角色绑定
[root@k8s-master ~]# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

  ###设置集群参数
  [root@k8s-master ~]# kubectl config set-cluster kubernetes \
    --certificate-authority=/opt/kubernetes/ssl/ca.pem \
    --embed-certs=true \
    --server=https://10.0.0.25:6443 \
    --kubeconfig=bootstrap.kubeconfig

  ###设置客户端认证参数
  [root@k8s-master ~]# kubectl config set-credentials kubelet-bootstrap \
     --token=30e6006d17ccc1ef829d26d1323cc7d6 \
     --kubeconfig=bootstrap.kubeconfig

  ###设置上下文参数
  [root@k8s-master ~]# kubectl config set-context default \
     --cluster=kubernetes \
     --user=kubelet-bootstrap \
     --kubeconfig=bootstrap.kubeconfig

###选择默认上下文
[root@k8s-master ssl]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig
[root@k8s-master ssl]# cp bootstrap.kubeconfig /opt/kubernetes/cfg
[root@k8s-master ssl]# scp bootstrap.kubeconfig 10.0.0.26:/opt/kubernetes/cfg  
[root@k8s-master ssl]# scp bootstrap.kubeconfig 10.0.0.27:/opt/kubernetes/cfg

###设置CNI支持   3个节点都创建
mkdir -p /etc/cni/net.d
vim /etc/cni/net.d/10-default.conf
{
        "name": "flannel",
        "type": "flannel",
        "delegate": {
            "bridge": "docker0",
            "isDefaultGateway": true,
            "mtu": 1400
        }
}
[root@k8s-master ~]# scp /etc/cni/net.d/10-default.conf 10.0.0.26:/etc/cni/net.d/10-default.conf 
[root@k8s-master ~]# scp /etc/cni/net.d/10-default.conf 10.0.0.27:/etc/cni/net.d/10-default.conf

###创建kubelet目录  #3个节点都创建
mkdir /var/lib/kubelet

###master节点配置文件
[root@k8s-master ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
  --address=10.0.0.25 \
  --hostname-override=10.0.0.25 \
  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/kubernetes/bin/cni \
  --cluster-dns=10.1.0.2 \
  --cluster-domain=cluster.local. \
  --hairpin-mode hairpin-veth \
  --allow-privileged=true \
  --fail-swap-on=false \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5

###node1节点配置文件
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
  --address=10.0.0.26 \
  --hostname-override=10.0.0.26 \
  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/kubernetes/bin/cni \
  --cluster-dns=10.1.0.2 \
  --cluster-domain=cluster.local. \
  --hairpin-mode hairpin-veth \
  --allow-privileged=true \
  --fail-swap-on=false \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5

###node2节点配置文件
[root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \
  --address=10.0.0.27 \
  --hostname-override=10.0.0.27 \
  --pod-infra-container-image=mirrorgooglecontainers/pause-amd64:3.0 \
  --experimental-bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
  --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
  --cert-dir=/opt/kubernetes/ssl \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/kubernetes/bin/cni \
  --cluster-dns=10.1.0.2 \
  --cluster-domain=cluster.local. \
  --hairpin-mode hairpin-veth \
  --allow-privileged=true \
  --fail-swap-on=false \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log
Restart=on-failure
RestartSec=5

###启动Kubelet   #3个节点都做
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

###查看csr请求
kubectl get csr

 

###批准kubelet的TLS证书请求
kubectl get csr|grep 'Pending' | awk 'NR>0{print $1}'| xargs kubectl certificate approve

###查看节点状态

kubectl get node

复制代码

4.2-kube-proxy部署

复制代码

###配置kube-proxy使用LVS
[root@k8s-master ~]#  yum install -y ipvsadm ipset conntrack
[root@k8s-node-1 ~]#  yum install -y ipvsadm ipset conntrack
[root@k8s-node-2 ~]#  yum install -y ipvsadm ipset conntrack

###创建 kube-proxy 证书请求
[root@k8s-master ~]# cd /usr/local/src/ssl/
[root@k8s-master ssl]# vim kube-proxy-csr.json
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

   
###生成证书
[root@k8s-master ssl]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes  kube-proxy-csr.json | cfssljson -bare kube-proxy

###分发证书到所有节点
[root@k8s-master ssl]# cp kube-proxy*.pem /opt/kubernetes/ssl/
[root@k8s-master ssl]# scp kube-proxy*.pem 10.0.0.26:/opt/kubernetes/ssl/
[root@k8s-master ssl]# scp kube-proxy*.pem 10.0.0.27:/opt/kubernetes/ssl/

###创建kube-proxy配置文件  #master节点做
[root@k8s-master ssl]# kubectl config set-cluster kubernetes \
   --certificate-authority=/opt/kubernetes/ssl/ca.pem \
   --embed-certs=true \
   --server=https://10.0.0.25:6443 \
   --kubeconfig=kube-proxy.kubeconfig
[root@k8s-master ssl]# kubectl config set-credentials kube-proxy \
   --client-certificate=/opt/kubernetes/ssl/kube-proxy.pem \
   --client-key=/opt/kubernetes/ssl/kube-proxy-key.pem \
   --embed-certs=true \
   --kubeconfig=kube-proxy.kubeconfig
[root@k8s-master ssl]# kubectl config set-context default \
   --cluster=kubernetes \
   --user=kube-proxy \
   --kubeconfig=kube-proxy.kubeconfig
[root@k8s-master ssl]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
[root@k8s-master ssl]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg/
[root@k8s-master ssl]# scp kube-proxy.kubeconfig 10.0.0.26:/opt/kubernetes/cfg/
[root@k8s-master ssl]# scp kube-proxy.kubeconfig 10.0.0.27:/opt/kubernetes/cfg/

###创建kube-proxy服务配置    #3个节点都创建
[root@k8s-master ssl]# mkdir /var/lib/kube-proxy
[root@k8s-node-1 ~]# mkdir /var/lib/kube-proxy
[root@k8s-node-2 ~]# mkdir /var/lib/kube-proxy
##master节点
[root@k8s-master ssl]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
  --bind-address=10.0.0.25 \
  --hostname-override=10.0.0.25 \
  --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
  --feature-gates=SupportIPVSProxyMode=true \
  --proxy-mode=ipvs \
  --ipvs-min-sync-period=5s \
  --ipvs-sync-period=5s \
  --ipvs-scheduler=rr \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
##node1节点
[root@k8s-node-1 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
  --bind-address=10.0.0.26 \
  --hostname-override=10.0.0.26 \
  --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
  --feature-gates=SupportIPVSProxyMode=true \
  --proxy-mode=ipvs \
  --ipvs-min-sync-period=5s \
  --ipvs-sync-period=5s \
  --ipvs-scheduler=rr \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
##node2节点
[root@k8s-node-2 ~]# vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \
  --bind-address=10.0.0.27 \
  --hostname-override=10.0.0.27 \
  --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig \
--masquerade-all \
  --feature-gates=SupportIPVSProxyMode=true \
  --proxy-mode=ipvs \
  --ipvs-min-sync-period=5s \
  --ipvs-sync-period=5s \
  --ipvs-scheduler=rr \
  --logtostderr=true \
  --v=2 \
  --logtostderr=false \
  --log-dir=/opt/kubernetes/log

Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

###启动Kubernetes Proxy    #3个节点都做
systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

检查LVS状态
ipvsadm -L -n
kubectl get node

复制代码

回到顶部

第五章：Flannel网络部署

5.1 部署Flanne

复制代码

###为Flannel生成证书
[root@k8s-master ~]# cd /usr/local/src/ssl/
[root@k8s-master ~]# vim flanneld-csr.json
{
  "CN": "flanneld",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

###生成证书
[root@k8s-master ~]# cfssl gencert -ca=/opt/kubernetes/ssl/ca.pem \
   -ca-key=/opt/kubernetes/ssl/ca-key.pem \
   -config=/opt/kubernetes/ssl/ca-config.json \
   -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

###分发证书
[root@k8s-master ssl]# cp flanneld*.pem /opt/kubernetes/ssl/
[root@k8s-master ssl]# scp flanneld*.pem 10.0.0.26:/opt/kubernetes/ssl/
[root@k8s-master ssl]# scp flanneld*.pem 10.0.0.27:/opt/kubernetes/ssl/

###下载Flannel软件包
[root@k8s-master ~]# cd /usr/local/src
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
[root@k8s-master src]# tar zxf flannel-v0.10.0-linux-amd64.tar.gz
[root@k8s-master src]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/
[root@k8s-master src]# scp flanneld mk-docker-opts.sh 10.0.0.26:/opt/kubernetes/bin/
[root@k8s-master src]# scp flanneld mk-docker-opts.sh 10.0.0.27:/opt/kubernetes/bin/

###复制对应脚本到/opt/kubernetes/bin目录
[root@k8s-master  ~]# cd /usr/local/src/kubernetes/cluster/centos/node/bin/
[root@k8s-master bin]# cp remove-docker0.sh /opt/kubernetes/bin/
[root@k8s-master bin]# scp remove-docker0.sh 10.0.0.26:/opt/kubernetes/bin/
[root@k8s-master bin]# scp remove-docker0.sh 10.0.0.27:/opt/kubernetes/bin/

###配置Flannel
[root@k8s-master  ~]# vim /opt/kubernetes/cfg/flannel
FLANNEL_ETCD="-etcd-endpoints=https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379"
FLANNEL_ETCD_KEY="-etcd-prefix=/kubernetes/network"
FLANNEL_ETCD_CAFILE="--etcd-cafile=/opt/kubernetes/ssl/ca.pem"
FLANNEL_ETCD_CERTFILE="--etcd-certfile=/opt/kubernetes/ssl/flanneld.pem"
FLANNEL_ETCD_KEYFILE="--etcd-keyfile=/opt/kubernetes/ssl/flanneld-key.pem"
##复制配置到其它节点上
[root@k8s-master  ~]# scp /opt/kubernetes/cfg/flannel 10.0.0.26:/opt/kubernetes/cfg/
[root@k8s-master  ~]# scp /opt/kubernetes/cfg/flannel 10.0.0.27:/opt/kubernetes/cfg/

###设置Flannel系统服务
[root@k8s-master  ~]# vim /usr/lib/systemd/system/flannel.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
Before=docker.service

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/flannel
ExecStartPre=/opt/kubernetes/bin/remove-docker0.sh
ExecStart=/opt/kubernetes/bin/flanneld ${FLANNEL_ETCD} ${FLANNEL_ETCD_KEY} ${FLANNEL_ETCD_CAFILE} ${FLANNEL_ETCD_CERTFILE} ${FLANNEL_ETCD_KEYFILE}
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -d /run/flannel/docker

Type=notify

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
##复制系统服务脚本到其它节点上
[root@k8s-master  ~]# scp /usr/lib/systemd/system/flannel.service 10.0.0.26:/usr/lib/systemd/system/
[root@k8s-master  ~]# scp /usr/lib/systemd/system/flannel.service 10.0.0.27:/usr/lib/systemd/system/

###下载Flannel CNI集成插件
[root@k8s-master  ~]# mkdir /opt/kubernetes/bin/cni   #3个节点都要创建
[root@k8s-master ~]# cd /usr/local/src/
[root@k8s-master src]# wget https://github.com/containernetworking/plugins/releases/download/v0.7.1/cni-plugins-amd64-v0.7.1.tgz
[root@k8s-master src]# tar zxf cni-plugins-amd64-v0.7.1.tgz -C /opt/kubernetes/bin/cni
[root@k8s-master src]# scp -r /opt/kubernetes/bin/cni/* 10.0.0.26:/opt/kubernetes/bin/cni/
[root@k8s-master src]# scp -r /opt/kubernetes/bin/cni/* 10.0.0.27:/opt/kubernetes/bin/cni/

###创建Etcd的key
[root@k8s-master  ~]# /opt/kubernetes/bin/etcdctl --ca-file /opt/kubernetes/ssl/ca.pem --cert-file /opt/kubernetes/ssl/flanneld.pem --key-file /opt/kubernetes/ssl/flanneld-key.pem \
      --no-sync -C https://10.0.0.25:2379,https://10.0.0.26:2379,https://10.0.0.27:2379 \
mk /kubernetes/network/config '{ "Network": "10.2.0.0/16", "Backend": { "Type": "vxlan", "VNI": 1 }}' >/dev/null 2>&1

###启动flannel   #3个节点都做
chmod +x /opt/kubernetes/bin/*
systemctl daemon-reload
systemctl enable flannel
systemctl start flannel
systemctl status flannel

###配置Docker使用Flannel
[root@k8s-master  ~]# vim /usr/lib/systemd/system/docker.service
#在Unit下面修改After和增加Requires
[Unit]   
After=network-online.target firewalld.service flannel.service
Requires=flannel.service

#增加EnvironmentFile=-/run/flannel/docker和$DOCKER_OPTS
[Service]  
EnvironmentFile=-/run/flannel/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

###将配置复制到另外两个阶段
[root@k8s-master  ~]# scp /usr/lib/systemd/system/docker.service 10.0.0.26:/usr/lib/systemd/system/
[root@k8s-master  ~]# scp /usr/lib/systemd/system/docker.service 10.0.0.27:/usr/lib/systemd/system/

###重启Docker    #3个节点都做
systemctl daemon-reload
systemctl restart docker

复制代码

5.2 创建一个K8S应用

复制代码

###创建一个测试用的deployment
[root@k8s-master ~]# kubectl run net-test --image=alpine --replicas=2 sleep 360000

###查看获取IP情况
[root@k8s-master ~]# kubectl get pod -o wide
NAME                        READY     STATUS    RESTARTS   AGE       IP          NODE
net-test-5767cb94df-7kqhz   1/1       Running   0          30m       10.2.93.2   10.0.0.27
net-test-5767cb94df-xw6r9   1/1       Running   0          30m       10.2.58.2   10.0.0.25

###测试联通性
[root@k8s-master ~]# ping 10.2.58.2 -c1

复制代码

如下图所示说明成功：

复制代码

###创建nginx的deployment.yaml文件
[root@k8s-master ~]# vim nginx-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.10.3
        ports:
        - containerPort: 80

###创建deployment        
[root@k8s-master ~]# kubectl create -f nginx-deployment.yaml 
deployment.apps "nginx-deployment" created

###查看deployment
[root@k8s-master ~]# kubectl get deployment
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
net-test           2         2         2            2           43m
nginx-deployment   3         3         3            0           53s
[root@k8s-master ~]# kubectl describe deployment nginx-deployment

###查看pod
[root@k8s-master ~]# kubectl get pod
NAME                                READY     STATUS              RESTARTS   AGE
net-test-5767cb94df-7kqhz           1/1       Running             0          45m
net-test-5767cb94df-xw6r9           1/1       Running             0          45m
nginx-deployment-75d56bb955-8tzh6   0/1       ContainerCreating   0          2m      #ContainerCreating说明正在创建中
nginx-deployment-75d56bb955-bjmg7   0/1       ContainerCreating   0          2m      #ContainerCreating说明正在创建中
nginx-deployment-75d56bb955-w4dks   0/1       ContainerCreating   0          2m      #ContainerCreating说明正在创建中

[root@k8s-master ~]# kubectl describe pod nginx-deployment-75d56bb955-8tzh6|tail -2
  Normal  SuccessfulMountVolume  5m    kubelet, 10.0.0.27  MountVolume.SetUp succeeded for volume "default-token-rpdp6"
  Normal  Pulling                5m    kubelet, 10.0.0.27  pulling image "nginx:1.10.3"         #正在pull取nginx的镜像

[root@k8s-master ~]# kubectl get pod -o wide
NAME                                READY     STATUS              RESTARTS   AGE       IP          NODE
net-test-5767cb94df-7kqhz           1/1       Running             0          49m       10.2.93.2   10.0.0.27
net-test-5767cb94df-xw6r9           1/1       Running             0          49m       10.2.58.2   10.0.0.25
nginx-deployment-75d56bb955-8tzh6   0/1       ContainerCreating   0          6m        <none>      10.0.0.27
nginx-deployment-75d56bb955-bjmg7   0/1       ContainerCreating   0          6m        <none>      10.0.0.26
nginx-deployment-75d56bb955-w4dks   0/1       ContainerCreating   0          6m        <none>      10.0.0.25


[root@k8s-master ~]# kubectl get pod -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP          NODE
net-test-5767cb94df-7kqhz           1/1       Running   0          1h        10.2.93.2   10.0.0.27
net-test-5767cb94df-xw6r9           1/1       Running   0          1h        10.2.58.2   10.0.0.25
nginx-deployment-75d56bb955-8tzh6   1/1       Running   0          21m       10.2.93.3   10.0.0.27
nginx-deployment-75d56bb955-bjmg7   1/1       Running   0          21m       10.2.83.2   10.0.0.26
nginx-deployment-75d56bb955-w4dks   1/1       Running   0          21m       10.2.58.3   10.0.0.25

###测试连通性
[root@k8s-master ~]# curl -I 10.2.93.3  -s|awk 'NR==1{print$2}'
200
[root@k8s-master ~]# curl -I 10.2.83.2  -s|awk 'NR==1{print$2}'
200
[root@k8s-master ~]# curl -I 10.2.58.3  -s|awk 'NR==1{print$2}'
200

###更新deployment
#--record是记录日志，方便回滚
[root@k8s-master ~]# kubectl set image deployment/nginx-deployment nginx=nginx:1.12.2 --record
deployment.apps "nginx-deployment" image updated


###查看更新后的deployment
[root@k8s-master ~]# kubectl get deployment -o wide
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES         SELECTOR
net-test           2         2         2            2           2h        net-test     alpine         run=net-test
nginx-deployment   3         3         3            3           1h        nginx        nginx:1.12.2   app=nginx

###查看更新历史
[root@k8s-master ~]# kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
  Labels:    app=nginx
    pod-template-hash=3181266511
  Containers:
   nginx:
    Image:    nginx:1.10.3
    Port:    80/TCP
    Host Port:    0/TCP
    Environment:    <none>
    Mounts:    <none>
  Volumes:    <none>


#查看具体某一个版本的更新历史
[root@k8s-master ~]# kubectl rollout history deployment/nginx-deployment --revision=1
deployments "nginx-deployment" with revision #1
Pod Template:
  Labels:    app=nginx
    pod-template-hash=3181266511
  Containers:
   nginx:
    Image:    nginx:1.10.3
    Port:    80/TCP
    Host Port:    0/TCP
    Environment:    <none>
    Mounts:    <none>
  Volumes:    <none>

###快速回滚到上一个版本
[root@k8s-master ~]# kubectl rollout undo deployment/nginx-deployment
deployment.apps "nginx-deployment"

[root@k8s-master ~]# vim nginx-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

###创建nginx的service.yaml文件
[root@k8s-master ~]# kubectl create -f nginx-service.yaml
service "nginx-service" created

###查看service
[root@k8s-master ~]# kubectl get service
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes      ClusterIP   10.1.0.1      <none>        443/TCP   18h
nginx-service   ClusterIP   10.1.172.14   <none>        80/TCP    12s

###测试连通性
[root@k8s-master ~]# curl --head http://10.1.172.14
HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Mon, 04 Jun 2018 04:58:47 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 31 Jan 2017 15:01:11 GMT
Connection: keep-alive
ETag: "5890a6b7-264"
Accept-Ranges: bytes

###查看虚IP
[root@k8s-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.0.1:443 rr persistent 10800
  -> 10.0.0.25:6443               Masq    1      0          0         
TCP  10.1.172.14:80 rr
  -> 10.2.58.5:80                 Masq    1      0          0         
  -> 10.2.83.4:80                 Masq    1      0          0         
  -> 10.2.93.5:80                 Masq    1      0          1 

###自动扩容到5个副本节点
[root@k8s-master ~]# kubectl scale deployment nginx-deployment --replicas 5
deployment.extensions "nginx-deployment" scaled
[root@k8s-master ~]# kubectl get pod -o wide
NAME                                READY     STATUS    RESTARTS   AGE       IP          NODE
net-test-5767cb94df-7kqhz           1/1       Running   0          2h        10.2.93.2   10.0.0.27
net-test-5767cb94df-xw6r9           1/1       Running   0          2h        10.2.58.2   10.0.0.25
nginx-deployment-75d56bb955-6dfxr   1/1       Running   0          17s       10.2.83.5   10.0.0.26
nginx-deployment-75d56bb955-9jmch   1/1       Running   0          17s       10.2.93.6   10.0.0.27
nginx-deployment-75d56bb955-gssxl   1/1       Running   0          7m        10.2.58.5   10.0.0.25
nginx-deployment-75d56bb955-hkqdc   1/1       Running   0          7m        10.2.93.5   10.0.0.27
nginx-deployment-75d56bb955-s66c4   1/1       Running   0          7m        10.2.83.4   10.0.0.26

[root@k8s-master ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.1.0.1:443 rr persistent 10800
  -> 10.0.0.25:6443               Masq    1      0          0         
TCP  10.1.172.14:80 rr
  -> 10.2.58.5:80                 Masq    1      0          0         
  -> 10.2.83.4:80                 Masq    1      0          0         
  -> 10.2.83.5:80                 Masq    1      0          0         
  -> 10.2.93.5:80                 Masq    1      0          0         
  -> 10.2.93.6:80                 Masq    1      0          0

复制代码

不知你与我能否走过这漫长岁月，又可否彼此陪伴于身边，或许因为不是同类的原因，总会有不同的性格

猜你喜欢

转载自blog.csdn.net/English0523/article/details/99417431
今日推荐
中国码农的“35岁魔咒”
蘭雅 CorelDRAW 插件 2024.5.1 国际劳动节版，免费下载
Arc Browser for Windows 1.0 正式 GA
90后程序员开发视频搬运软件、不到一年获利超 700 万，结局很刑！
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
周排行
Java基础复习_day13_Collection集合
2018.11.16 c语言学习经验
且看Java内置四大核心函数式接口
小程序云开发中数据库的数据分段和显示图片
python的函数
Web-JS进阶
【干货】C++常用代码积累笔记大全
Spring的ioc操作 与 IOC底层原理
构建之法20191121-11 Scrum立会报告+燃尽图 07
Spring boot之Hello World访问404
每日归档
更多
2024-05-05(0)
2024-05-04(7)
2024-05-03(19)
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022