第一章 K8s安装
1.1 基础环境配置
参考：
https://www.jianshu.com/p/832bcd89bc07
https://www.cnblogs.com/ericnie/p/7749588.html
http://www.cnblogs.com/cuibobo/articles/8276291.html
1.1.1 $ cat /etc/hosts
192.168.11.1 master
192.168.11.2 node
1.1.2 禁用防火墙
查看防火墙状态
$ firewall-cmd --state
关闭防火墙
$ systemctl stop firewalld
$ systemctl disable firewalld
1.1.3 关闭Swap
运行cat /proc/swaps 检查果然Swap没有被关闭。
# 关闭Swap，机器重启后不生效
swapoff -a
# 修改/etc/fstab永久关闭Swap
cp -p /etc/fstab /etc/fstab.bak$(date '+%Y%m%d%H%M%S')
# Redhat
sed -i "s/\/dev\/mapper\/rhel-swap/\#\/dev\/mapper\/rhel-swap/g" /etc/fstab
# CentOS
sed -i "s/\/dev\/mapper\/centos-swap/\#\/dev\/mapper\/centos-swap/g" /etc/fstab
# 修改后重新挂载全部挂载点
mount -a
# 查看Swap
free -m
cat /proc/swaps
以redhat为例：
1.1.3.1 以redhat系统为例
1) 确定系统名称
$ uname #系统名称
2) 确定具体系统
$ cat /etc/redhat-release #具体系统
3) 检查Swaps是否已关闭
$ cat /proc/swaps
未关闭
已关闭
4) 关闭swaps步骤
$ swapoff –a #暂时关闭swap（重启后不生效），永久关闭执行下面两条命令，需要重启生效
$ cp -p /etc/fstab /etc/fstab.bak$(date '+%Y%m%d%H%M%S')
$ sed -i "s/\/dev\/mapper\/rhel-swap/\#\/dev\/mapper\/rhel-swap/g" /etc/fstab
$ mount -a
1.1.4 禁用SELINUX
$ setenforce 0
$ cat /etc/selinux/config
SELINUX=disabled
1.1.5 创建/etc/sysctl.d/k8s.conf文件，添加如下内容
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
执行如下命令使修改生效：
$ modprobe br_netfilter
$ sysctl -p /etc/sysctl.d/k8s.conf
1.2 安装 kubeadm、kubelet、kubectl
1.2.1 安装
$ yum install -y kubelet-1.11.0-0.x86_64.rpm kubeadm-1.11.0-0.x86_64.rpm kubectl-1.11.0-0.x86_64.rpm kubernetes-cni-0.6.0-0.x86_64.rpm cri-tools-1.11.0-0.x86_64.rpm socat-1.7.1.3-1.el6.rf.x86_64.rpm
1.2.2 配置kubelet
安装完成后，我们还需要对kubelet进行配置，因为用yum源的方式安装的kubelet生成的配置文件将参数--cgroup-driver改成了systemd，而docker的cgroup-driver是cgroupfs，这二者必须一致才行，我们可以通过docker info命令查看：
$ docker info |grep Cgroup
Cgroup Driver: cgroupfs
修改文件kubelet的配置文件/etc/systemd/system/kubelet.service.d/10-kubeadm.conf，将其中的KUBELET_CGROUP_ARGS参数更改成cgroupfs：
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
另外还有一个问题是关于交换分区的，之前我们在手动搭建高可用的kubernetes 集群一文中已经提到过，Kubernetes从1.8开始要求关闭系统的 Swap ，如果不关闭，默认配置的kubelet将无法启动，我们可以通过 kubelet 的启动参数--fail-swap-on=false更改这个限制，所以我们需要在上面的配置文件中增加一项配置(在ExecStart之前)：
Environment="KUBELET_EXTRA_ARGS=--fail-swap-on=false"
当然最好的还是将swap给关掉，这样能提高kubelet的性能。修改完成后，重新加载我们的配置文件即可：
$ systemctl daemon-reload
1.2.3 安装k8相关镜像
docker load --input etcd-amd64_3.2.18.tar
docker load --input k8s-dns-kube-dns-amd64_1.14.8.tar
docker load --input kube-controller-manager-amd64_v1.11.0.tar
docker load --input pause_3.1.tar
docker load --input flannel_v0.10.0-amd64.tar
docker load --input k8s-dns-sidecar-amd64_1.14.8.tar
docker load --input kube-proxy-amd64_v1.11.0.tar
docker load --input k8s-dns-dnsmasq-nanny-amd64_1.14.8.tar
docker load --input kube-apiserver-amd64_v1.11.0.tar
docker load --input kube-scheduler-amd64_v1.11.0.tar
docker load --input pause-amd64_3.1.tar
docker load --input coredns_1.1.3.tar
1.3 初始化 Master 节点
kubeadm init --kubernetes-version=1.11.0 --apiserver-advertise-address 172.22.1.185 --pod-network-cidr=10.244.0.0/16
I0905 16:01:50.462186 21858 feature_gate.go:230] feature gates: &{map[]}
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
I0905 16:01:50.481910 21858 kernel_validator.go:81] Validating kernel version
I0905 16:01:50.481992 21858 kernel_validator.go:96] Validating kernel config
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.05.0-ce. Max validated version: 17.03
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [docker185 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.22.1.185]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [docker185 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [docker185 localhost] and IPs [172.22.1.185 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 41.002557 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node docker185 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node docker185 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "docker185" as an annotation
[bootstraptoken] using token: qepaue.2rj1lsdt6jxr0q8z
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 172.22.1.185:6443 --token qepaue.2rj1lsdt6jxr0q8z --discovery-token-ca-cert-hash sha256:6c54033e82ac761333c37505df625c924524ce2adfd9afc1af6893ae5b1d70bb
kubeadm会自动检查当前环境是否有上次命令执行的“残留”。如果有，必须清理后再行执行init。我们可以通过”kubeadm reset”来清理环境，以备重来。
centos 系统的服务器打开终端很慢，并显示'abrt-cli status' timed out
输入abrt-auto-reporting enabled可解决
1.3.1 配置kubectl认证信息（Master节点操作）
# 对于非root用户
su -ubuntu
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 对于root用户
export KUBECONFIG=/etc/kubernetes/admin.conf
也可以直接放到~/.bash_profile
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
1.3.2 启动kubelet
sudo systemctl enable kubelet && sudo systemctl start kubelet
1.3.3 重启kubelet
systemctl daemon-reload
systemctl restart kubelet
1.3.4 检查kubelet是否启动成功
sudo systemctl status kubelet.service
1.3.5 为了使用更便捷，启用kubectl命令的自动补全功能：
echo "source <(kubectl completion bash)" >> ~/.bashrc
1.3.6 查看版本
$ kubectl version
问题
运行kubectl version
信息显示8080端口无法找到
解决方案：
alias kubectl='kubectl --kubeconfig=/etc/kubernetes/kubelet.kubeconfig'
unalias kubectl
https://yq.aliyun.com/articles/149595
kubectl proxy --port=8080 &
1.4 安装pod网络
1.4.1 获取pod
export KUBECONFIG=/etc/kubernetes/admin.conf
[root@k8s-1 ~]# kubectl get pods --all-namespaces
1.5 安装pod网络
要让kubernetes cluster能够工作，必须安装pod网络，否则pod之间无法通讯。Kubernetes支持多种网络方案，这里我们先使用flannel。
kubectl create -f kube-flannel-rbac.yml
kubectl create -f kube-flannel.yml
查看节点状态：
[root@localhost images]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
localhost Ready master 2h v1.11.0
默认配置下 Kubernetes 不会将 Pod 调度到 Master 节点。如果希望将 k8s-master 也当作 Node 使用，可以执行如下命令：
kubectl taint node localhost node-role.kubernetes.io/master-
kubectl taint node docker185 node-role.kubernetes.io/master-
例子：
[root@localhost images]# kubectl taint node localhost node-role.kubernetes.io/master-
node/localhost untainted
如果要恢复 Master Only 状态，执行如下命令：
kubectl taint node localhost node-role.kubernetes.io/master="":NoSchedule
k8使用和维护
1.1 kubectl 命令
kubectl controls the Kubernetes cluster manager
| 命令 |
描述 |
| Basic Commands (Beginner): |
|
| create |
Create a resource from a file or from stdin. |
| expose |
用 replication controller, service, deployment 或者 pod |
| 并暴露它作为一个 新的 Kubernetes Service |
|
| run |
在集群中运行一个指定的镜像 |
| set |
为 objects 设置一个指定的特征 |
| Basic Commands (Intermediate): |
|
| explain |
查看资源的文档 |
| get |
显示一个或更多 resources |
| edit |
在服务器上编辑一个资源 |
| delete |
Delete resources by filenames, stdin, resources and names, or by resources and label selector |
| Deploy Commands: |
|
| rollout |
Manage the rollout of a resource |
| scale |
为 Deployment, ReplicaSet, Replication Controller 或者 Job |
| autoscale |
自动调整一个 Deployment, ReplicaSet, 或者 ReplicationController的副本数量 |
| Cluster Management Commands: |
|
| certificate |
修改 certificate 资源. |
| cluster-info |
显示集群信息 |
| top |
Display Resource (CPU/Memory/Storage) usage. |
| cordon |
标记 node 为 unschedulable |
| uncordon |
标记 node 为 schedulable |
| drain |
Drain node in preparation for maintenance |
| taint |
更新一个或者多个 node 上的 taints |
| Troubleshooting and Debugging Commands: |
|
| describe |
显示一个指定 resource 或者 group 的 resources 详情 |
| logs |
输出容器在 pod 中的日志 |
| attach |
Attach 到一个运行中的 container |
| exec |
在一个 container 中执行一个命令 |
| port-forward |
Forward one or more local ports to a pod |
| proxy |
运行一个 proxy 到 Kubernetes API server |
| cp |
复制 files 和 directories 到 containers 和从容器中复制 files 和 directories. |
| auth |
Inspect authorization |
| Advanced Commands: |
|
| apply |
通过文件名或标准输入流(stdin)对资源进行配置 |
| patch |
用 strategic merge patch 更新一个资源的 field(s) |
| replace |
通过 filename 或者 stdin替换一个资源 |
| wait |
Experimental: Wait for one condition on one or many resources |
| convert |
在不同的 API versions 转换配置文件 |
| Settings Commands: |
|
| label |
更新在这个资源上的 labels |
| annotate |
更新一个资源的注解 |
| completion |
Output shell completion code for the specified shell (bash or zsh) |
| Other Commands: |
|
| alpha |
Commands for features in alpha |
| api-resources |
Print the supported API resources on the server |
| api-versions |
Print the supported API versions on the server, in the form of "group/version" |
| config |
修改 kubeconfig 文件 |
| plugin |
Runs a command-line plugin |
| version |
输出 client 和 server 的版本信息 |
1.1.1 kubectl label
更新在这个资源上的 labels
用label控制pod的位置
kubectl label nodes localhost nd=web
有了nd这个自定义label，接下来就可以指定将pod部署到localhost节点上。编辑nginx.yml
[root@localhost yml]# vi nginx.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-deployment001
spec:
replicas: 5
template:
metadata:
labels:
app: web_server
spec:
containers:
- name: nginx
image: nginx:1.7.9
nodeSelector:
nd: web
要删除label nd 执行如下命令
kubectl label nodes localhost nd-
- 即删除
1.1.2 kubectl delete
Usage:
kubectl delete ([-f FILENAME] | TYPE [(NAME | -l label | --all)]) [options]
| docker delete |
|
| --all=false |
Delete all resources, including uninitialized ones, in the namespace of the specified resource types. |
| --cascade=true |
If true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a ReplicationController). Default true. |
| --field-selector='' |
Selector (field query) to filter on, supports '=', '==', and '!='.(e.g. --field-selector key1=value1,key2=value2). The server only supports a limited number of field queries per type. |
| -f, --filename=[] |
containing the resource to delete. |
| --force=false |
Only used when grace-period=0. If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation. |
| --grace-period=-1 |
Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion). |
| --ignore-not-found=false |
Treat "resource not found" as a successful delete. Defaults to "true" when --all is specified. |
| --include-uninitialized=false |
If true, the kubectl command applies to uninitialized objects. If explicitly set to false, this flag overrides other flags that make the kubectl commands apply to uninitialized objects, e.g., "--all". Objects with empty metadata.initializers are regarded as initialized. |
| --now=false |
If true, resources are signaled for immediate shutdown (same as --grace-period=1). |
| -o, --output='' |
Output mode. Use "-o name" for shorter output (resource/name). -R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory. |
| -l, --selector='' |
Selector (label query) to filter on, not including uninitialized ones. --timeout=0s: The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object |
| --wait=true: |
If true, wait for resources to be gone before returning. This waits for finalizers. |
1.1.2.1 删除pod
kubectl delete –f nginx.yml