kubernetes 1.9 installation and deployment

[TOC]

introduction

Provides tools for rapidly deploying high-availability k8s clusters, deploys based on binary mode and uses ansible-playbook to achieve automation. It not only provides one-click installation scripts, but also installs each component step by step, and explains the main parameter configuration and precautions of each step.

Features

Cluster features: TLS mutual authentication, RBAC authorization, multi-Master high availability, and support for Network Policy.

threshold

You need to master the basic knowledge of kubernetes docker linux shell. For ansible, it is recommended to read ansible ultra-fast introduction.

Environmental preparation

Node information

Node info/hostname	Intranet IP	install software
master1	192.168.16.8	ansible+calico+API server+scheduler+controller manager+etcd
master2	192.168.16.9	calico+API server+scheduler+controller manager+etcd
master3	192.168.16.15	calico+API server+scheduler+controller manager+etcd
node1	192.168.16.10	calico + kubelet + be proxy
node2	192.168.16.11	calico + kubelet + be proxy
node3	192.168.16.12	calico + kubelet + be proxy
load balancing intranet	192.168.16.16
harbor host	192.168.16.3	harbor

In order to save resources, in this deployment, the master is also used as the etcd deployment machine.
Create an LB in the VPC and use the LB-IP:portform as the kube-apiserver address to form the internal API interface of the multi-master cluster.
The harbor host can use the harbor deployed before.
The creation of cloud hosts for each node will not be described in detail.

Compute Node Specifications

In the test environment, all 6 nodes can use 2C 4G 40G default disks.

Architecture diagram

k8s architecture diagram

Install

load balancing information

Public clouds already have load balancing services in place of haproxy+keepalivedthe solution. The following is the load balancing and related information:

Virtual Server Group Information:

Monitoring rule information:

master1 node operation

Through ansible installation, you can operate on any one of the 6 nodes. In this article, the operation is performed on the master1 node.

Install dependencies and ansible

The version of ansbile is recommended to be above 2.4, otherwise it will report that the module cannot be recognized.

apt-get update && apt-get upgrade -y
apt-get install python2.7 git python-pip
pip install pip --upgrade
pip install ansible

Install the python encryption module

pip install cryptography --upgrade
#重新安装pyopenssl，否则会报错
pip uninstall pyopenssl
pip install pyopenssl

Configure ansible ssh keys

ssh-keygen -t rsa -b 2048  #3个回车
ssh-copy-id $IP  #$IP为本虚机地址，按照提示输入yes 和root密码，将密钥发送至各节点（包括本机内网IP）

Operation of each node

Basic software installation

apt-get update && apt-get upgrade -y && apt-get dist-upgrade -y
apt-get purge ufw lxd lxd-client lxcfs lxc-common -y

DNS settings

To prevent /etc/resolv.confDNS from being rewritten, write the DNS address to /etc/resolvconf/resolv.conf.d/base:

cat << EOF >> /etc/resolvconf/resolv.conf.d/base
nameserver 103.224.222.222
nameserver 103.224.222.223
nameserver 8.8.8.8
EOF

Modify hostname

hostname分为三种：

pretty hostname: 也就是比较好看的hostname，用来取悦自己的;)，如设置为“Zhjwpku’s Laptop”
static hostname: 用来在启动的时候初始化内核的hostname，保存在/etc/hostname中
transient hostname: 瞬态的主机名，是系统运行时临时分配的主机名，例如使用hostname node1 设置的主机名node1就为transient hostname

Use hostnamectl to set the above three host names. If not specified, static and transient are set at the same time by default.

Set the corresponding hostname on each node:

#$HostNames是各节点对应的主机名称
hostnamectl set-hostname $HostName

Modify the /etc/hosts file

All nodes add node information to /etc/hosts:

cat <<EOF >> /etc/hosts
192.168.16.8 master1
192.168.16.9 master2
192.168.16.15 master3
192.168.16.10 node1
192.168.16.11 node2
192.168.16.12 node3
192.168.16.3 harbor.jdpoc.com
EOF

configure ansible

Installation and deployment operations can be performed on the master1 node, and ansible script can be used to execute with one click.

Download the ansible template along with the K8S 1.9.6 binaries and extract them:

cd ~
wget http://chengchen.oss.cn-north-1.jcloudcs.com/ansible.tar.gz
wget http://chengchen.oss.cn-north-1.jcloudcs.com/k8s.196.tar.gz
tar zxf k8s.196.tar.gz
tar zxf ansible.tar.gz
#将bin目录中的文件移动至ansible/bin目录
mv bin/* ansible/bin/
#移动ansible目录至/etc
mv ansible /etc/

Edit ansible's configuration file.

cd /etc/ansible
cp example/hosts.m-masters.example hosts
vi hosts

Modified according to the actual situation, this deployment is configured as follows:

# 部署节点：运行这份 ansible 脚本的节点
# 实际情况修改
[deploy]
192.168.16.8

# etcd集群请提供如下NODE_NAME、NODE_IP变量,注意etcd集群必须是1,3,5,7...奇数个节点
# 实际情况修改
[etcd]
192.168.16.8 NODE_NAME=etcd1 NODE_IP="192.168.16.8"
192.168.16.9 NODE_NAME=etcd2 NODE_IP="192.168.16.9"
192.168.16.15 NODE_NAME=etcd3 NODE_IP="192.168.16.15"

[kube-master]
# 实际情况修改
192.168.16.8 NODE_IP="192.168.16.8"
192.168.16.9 NODE_IP="192.168.16.9"
192.168.16.15 NODE_IP="192.168.16.15"


####################在公有云环境中，可使用负载均衡，无需部署####################
# 负载均衡至少两个节点，安装 haproxy+keepalived
#[lb]
#192.168.1.1 LB_IF="eth0" LB_ROLE=backup  # 注意根据实际使用网卡设置 LB_IF变量
#192.168.1.2 LB_IF="eth0" LB_ROLE=master
#[lb:vars]
#master1="192.168.1.1:6443" # 根据实际master节点数量设置
#master2="192.168.1.2:6443" # 需同步设置roles/lb/templates/haproxy.cfg.j2 
#master3="192.168.1.x:6443"
#ROUTER_ID=57                    # 取值在0-255之间，区分多个instance的VRRP组播，同网段不能重复
#MASTER_PORT="8443"     # 设置 api-server VIP地址的服务端口
#################################################################################

# 实际情况修改
[kube-node]
192.168.16.10 NODE_IP="192.168.16.10"
192.168.16.11 NODE_IP="192.168.16.11"
192.168.16.12 NODE_IP="192.168.16.12"

# 如果启用harbor，请配置后面harbor相关参数，如果已有harbor，注释即可
[harbor]
#192.168.1.8 NODE_IP="192.168.1.8"

# 预留组，后续添加master节点使用，无预留注释即可
[new-master]
#192.168.1.5 NODE_IP="192.168.1.5"

# 预留组，后续添加node节点使用，无预留注释即可
[new-node]
#192.168.1.xx NODE_IP="192.168.1.xx"

[all:vars]
# ---------集群主要参数---------------
#集群部署模式：allinone, single-master, multi-master
# 根据实际情况选择：单机部署、单master、多master
DEPLOY_MODE=multi-master

#集群 MASTER IP即 LB节点VIP地址，并根据 LB节点的MASTER_PORT组成 KUBE_APISERVER
# 根据我们之前创建的负载均衡，填写内网IP即可，端口可自定义
MASTER_IP="192.168.16.16"
KUBE_APISERVER="https://192.168.16.16:8443"

#TLS Bootstrapping 使用的 Token，使用 head -c 16 /dev/urandom | od -An -t x | tr -d ' ' 生成
# 在系统中执行以上命令，生成结果替换下面变量即可
BOOTSTRAP_TOKEN="a7383de6fdf9a8cb661757c7b763feb6"

# 集群网络插件，目前支持calico和flannel
# 本次部署使用了calico
CLUSTER_NETWORK="calico"

# 部分calico相关配置，更全配置可以去roles/calico/templates/calico.yaml.j2自定义
# 设置 CALICO_IPV4POOL_IPIP=“off”,可以提高网络性能，条件限制详见 05.安装calico网络组件.md
CALICO_IPV4POOL_IPIP="always"
# 设置 calico-node使用的host IP，bgp邻居通过该地址建立，可手动指定端口"interface=eth0"或使用如下自动发现
# 公有云默认即可
IP_AUTODETECTION_METHOD="can-reach=223.5.5.5"

# 部分flannel配置，详见roles/flannel/templates/kube-flannel.yaml.j2
FLANNEL_BACKEND="vxlan"

# 服务网段 (Service CIDR），部署前路由不可达，部署后集群内使用 IP:Port 可达
SERVICE_CIDR="10.68.0.0/16"

# POD 网段 (Cluster CIDR），部署前路由不可达，**部署后**路由可达
CLUSTER_CIDR="172.21.0.0/16"

# 服务端口范围 (NodePort Range)
NODE_PORT_RANGE="20000-40000"

# kubernetes 服务 IP (预分配，一般是 SERVICE_CIDR 中第一个IP)
CLUSTER_KUBERNETES_SVC_IP="10.69.0.1"

# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_DNS_SVC_IP="10.69.0.2"

# 集群 DNS 域名
CLUSTER_DNS_DOMAIN="cluster.local."

# etcd 集群间通信的IP和端口, **根据实际 etcd 集群成员设置**
ETCD_NODES="etcd1=https://192.168.16.8:2380,etcd2=https://192.168.16.9:2380,etcd3=https://192.168.16.15:2380"

# etcd 集群服务地址列表, **根据实际 etcd 集群成员设置**
ETCD_ENDPOINTS="https://192.168.16.8:2379,https://192.168.16.9:2379,https://192.168.16.15:2379"

# 集群basic auth 使用的用户名和密码
BASIC_AUTH_USER="admin"
BASIC_AUTH_PASS="jdtest1234"

# ---------附加参数--------------------
#默认二进制文件目录
bin_dir="/root/local/bin"

#证书目录
ca_dir="/etc/kubernetes/ssl"

#部署目录，即 ansible 工作目录，建议不要修改
base_dir="/etc/ansible"

#私有仓库 harbor服务器 (域名或者IP)，如已有harbor，注释即可
#HARBOR_IP="192.168.16.3"
#HARBOR_DOMAIN="harbor.jdpoc.com"

Quick install kubernetes 1.9

Choose one of the following two installation modes to install.

Step-by-step installation

Execute the 01-06 yaml files in order:

cd /etc/ansible
ansible-playbook 01.prepare.yml
ansible-playbook 02.etcd.yml 
ansible-playbook 03.docker.yml
ansible-playbook 04.kube-master.yml
ansible-playbook 05.kube-node.yml
ansible-playbook 06.network.yml

If there is an error, verify the configuration in the file against the actual error message /etc/ansible/hosts.

One step installation

cd /etc/ansible
ansible-playbook 90.setup.yml

clean up

Optionally clean up installed programs in case of deployment errors:

cd /etc/ansible
ansible-playbook 99.clean.yml

Deploy Cluster DNS

DNS is the first thing that needs to be deployed in the k8s cluster. Other pods in the cluster use it to provide domain name resolution services; it can mainly resolve the cluster service name SVCand Pod hostname; currently, k8s v1.9+ version can have two choices: kube-dnsand coredns, you can choose one of them to deploy and install . used in this deployment kubedns.

kubectl create -f /etc/ansible/manifests/kubedns

The cluster pod inherits the dns parsing of the node by default, modify the kubelet service startup parameters --resolv-conf="", you can change this feature, see kubelet startup parameters for details
If you use the calico network component and install the dns component directly, the following bugs may occur. The analysis is because when calico allocates the pod address, it will start from the first address (network address) of the network segment. The temporary solution is to manually delete the POD and recreate it. After obtaining the following IP address

# BUG出现现象
$ kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                       READY     STATUS             RESTARTS   AGE       IP              NODE
default       busy-5cc98488d4-s894w                      1/1       Running            0          28m       172.20.24.193   192.168.97.24
kube-system   calico-kube-controllers-6597d9c664-nq9hn   1/1       Running            0          1h        192.168.97.24   192.168.97.24
kube-system   calico-node-f8gnf                          2/2       Running            0          1h        192.168.97.24   192.168.97.24
kube-system   kube-dns-69bf9d5cc9-c68mw                  0/3       CrashLoopBackOff   27         31m       172.20.24.192   192.168.97.24

# 解决办法，删除pod，自动重建
$ kubectl delete pod -n kube-system kube-dns-69bf9d5cc9-c68mw

Verify DNS service

Create a new test nginx service

kubectl run nginx --image=nginx --expose --port=80

Confirm nginx service:

root@master1:/etc/ansible/manifests/kubedns# kubectl get pod
NAME                     READY     STATUS    RESTARTS   AGE
nginx-7587c6fdb6-vjnss   1/1       Running   0          30m

Test pod busybox:

root@master1:/etc/ansible/manifests/kubedns# kubectl run busybox --rm -it --image=busybox /bin/sh
If you don't see a command prompt, try pressing enter.
/ # cat /etc/resolv.conf 
nameserver 10.69.0.2
search default.svc.cluster.local. svc.cluster.local. cluster.local.
options ndots:5
/ # nslookup nginx
Server:    10.69.0.2
Address 1: 10.69.0.2 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.69.152.34 nginx.default.svc.cluster.local
/ # nslookup www.baidu.com
Server:    10.69.0.2
Address 1: 10.69.0.2 kube-dns.kube-system.svc.cluster.local

Name:      www.baidu.com
Address 1: 220.181.112.244
Address 2: 220.181.111.188

If the nginx and external domain names can be successfully resolved, the DNS deployment is successful. If it cannot be resolved, there is a problem with kube-dns. Please pass kubectl get pod --all-namespaces -o wideto obtain the node where kube-dns is located, and by docker logsviewing the detailed logs, there is a high probability that it is due to Caused by the bug mentioned above.

Deploy the dashboard

deploy:

kubectl create -f /etc/ansible/manifests/dashboard/kubernetes-dashboard.yaml
#可选操作：部署基本密码认证配置，密码文件位于 /etc/kubernetes/ssl/basic-auth.csv
kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin
kubectl create -f /etc/ansible/manifests/dashboard/ui-admin-rbac.yaml

verify:

# 查看pod 运行状态
kubectl get pod -n kube-system | grep dashboard
kubernetes-dashboard-7c74685c48-9qdpn   1/1       Running   0          22s
# 查看dashboard service
kubectl get svc -n kube-system|grep dashboard
kubernetes-dashboard   NodePort    10.68.219.38   <none>        443:24108/TCP                   53s
# 查看集群服务，获取访问地址
root@master1:~# kubectl cluster-info
Kubernetes master is running at https://192.168.16.16:8443
# 此处就是我们要获取到的dashboard的URL地址。
kubernetes-dashboard is running at https://192.168.16.16:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
# 查看pod 运行日志，关注有没有错误
kubectl logs kubernetes-dashboard-7c74685c48-9qdpn -n kube-system

When accessing the page, replace the IP address in the access URL with the external network IP address of the load balancer. After opening the page, the following information will be displayed:

Get access token:

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

Open the page and the landing page that comes with the new version of the dashboard appears. We choose the "Token" method to log in, paste the obtained token, and click to log in:

Deploy heapster

The process of Heapster monitoring the resources of the entire cluster: First, the built-in cAdvisor of kubelet collects the container resource usage of the node node, then heapster collects the resource usage of nodes and containers from the API provided by kubelet, and finally heapster persists data and stores it in influxdb (or are other storage backends, Google Cloud Monitoring, etc.).

Grafana points to the above influxdb by configuring the data source to display monitoring information in an interface.

deploy

Deployment is very simple, just execute the following command:

kubectl create -f /etc/ansible/manifests/heapster/

verify

root@master1:~# kubectl get pods -n kube-system |grep -E "heapster|monitoring"
heapster-7f8bf9bc46-w6xbr                  1/1       Running   0          2d
monitoring-grafana-59c998c7fc-gks5j        1/1       Running   0          2d
monitoring-influxdb-565ff5f9b6-xth2x       1/1       Running   0          2d

View log:

kubectl logs heapster-7f8bf9bc46-w6xbr -n kube-system 
kubectl logs monitoring-grafana-59c998c7fc-gks5j -n kube-system
kubectl logs monitoring-influxdb-565ff5f9b6-xth2x -n kube-system

visit grafana

#获取grafana的URL连接
root@master1:~# kubectl cluster-info | grep grafana
monitoring-grafana is running at https://192.168.16.16:8443/api/v1/namespaces/kube-system/services/monitoring-grafan/proxy

Open the connection:

You can see the CPU, memory, load and other utilization graphs of each Nodes and Pods. If the utilization graph cannot be seen on the dashboard, use the following command to restart the dashboard pod:

delete firstkubectl scale deploy kubernetes-dashboard --replicas=0 -n kube-system
then create a newkubectl scale deploy kubernetes-dashboard --replicas=1 -n kube-system

After deploying heapster, use the kubectl client tool to view resource usage directly

# 查看node 节点资源使用情况
$ kubectl top node  
# 查看各pod 的资源使用情况
$ kubectl top pod --all-namespaces