Docker (twenty-one)--Docker k8s--Kubernetes storage--kubernetes monitoring--Helm

1 Introduction

1.1. Introduction

• Helm is a package management tool for Kubernetes applications, mainly used to manage Charts, similar to yum in Linux systems.

• Helm Chart is a series of YAML files used to package Kubernetes native applications. You can customize some metadata of the application when you deploy the application to facilitate the distribution of the application.

• For application publishers, Helm can package applications, manage application dependencies, manage application versions, and publish applications to software warehouses.

• For users, there is no need to write complex application deployment files after using Helm, and applications can be found, installed, upgraded, rolled back, and uninstalled on Kubernetes in a simple way.

1.2 The biggest difference between Helm V3 and V2 is the elimination of tiller

2. Configuration

2.1 Pre-configuration

Helm official website

• 1. Helm installation:
  download the software package: helm-v3.1.1-linux-amd64.tar.gz
  $ tar zxf helm-v3.1.1-linux-amd64.tar.gz
  $ cd linux-amd64/
  $ cp helm /usr/ local/bin/

• 2. Set the helm command to complete:
  echo "source <(helm completion bash)" >> ~/.bashrc

• 3. Search the official helm hub chart library:
  $ helm search hub wordpress

• 4. Helm to add a third-party Chart library:
  $ helm repo add stable http://mirror.azure.cn/kubernetes/charts/
  $ helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
  $ helm search repo redis

• 4. Helm deployment application:
  $ $ helm search repo redis //Query
  NAME CHART VERSION APP VERSION
  stable/redis 10.5.6 5.0.7
  stable/redis-ha 4.4.0 5.0.6
  …
  support multiple installation methods: (helm default Read ~/.kube/config information to connect to k8s cluster)
  $ helm install redis-ha stable/redis-ha
  $ helm install redis-ha redis-ha-4.4.0.tgz
  $ helm install redis-ha path/redis-ha
  $ helm install redis-ha https://example.com/charts/redis-ha-4.4.0.tgz
  $ helm pull stable/redis-ha // pull the application to the local
  $ helm status redis-ha // view status
  $ helm uninstall redis-ha //uninstall

[root@server2 ~]# mkdir helm
[root@server2 ~]# cd helm/
[root@server2 helm]# ls
helm-v3.4.1-linux-amd64.tar.gz
[root@server2 helm]# tar zxf helm-v3.4.1-linux-amd64.tar.gz  
[root@server2 helm]# ll
total 13012
-rwxr-xr-x 1 root root 13323294 Mar  3 16:09 helm-v3.4.1-linux-amd64.tar.gz
drwxr-xr-x 2 3434 3434       50 Nov 12 03:52 linux-amd64
[root@server2 helm]# cd linux-amd64/
[root@server2 linux-amd64]# ls
helm  LICENSE  README.md
[root@server2 linux-amd64]# cp helm /usr/local/bin/   ##设置环境变量
[root@server2 linux-amd64]# helm env     ##查看环境变量
[root@server2 linux-amd64]# helm list     ##查看应用
[root@server2 ~]# echo "source <(helm completion bash)" >> ~/.bashrc  ##设置补齐命令
[root@server2 ~]# cat .bashrc      ##查看环境变量
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi
source <(kubectl completion bash)
source <(helm completion bash)
[root@server2 ~]# source .bashrc    ##更新环境变量
[root@server2 ~]# helm search hub redis-ha   ##查找官方库

[root@server2 ~]# helm repo add stable http://mirror.azure.cn/kubernetes/charts/  ##添加库，作为测试
[root@server2 ~]# helm repo remove stable    ##测试完成删除即可
[root@server2 ~]# helm repo add bitnami https://charts.bitnami.com/bitnami
[root@server2 ~]# helm search repo redis-ha     ##测试一下
[root@server2 ~]# helm search repo metrics-server   ##测试

2.2 Experiment

## 1. 配置redis镜像
[root@server1 harbor]# docker pull redis:6.0.7
[root@server1 harbor]# docker tag redis:6.0.7 reg.westos.org/library/redis:6.0.7-alpine 
[root@server1 harbor]# docker push reg.westos.org/library/redis:6.0.7-alpine 

## 2. 配置
[root@server2 ~]# helm repo add dandydev https://dandydeveloper.github.io/charts  ##官网上搜索redis-ha，然后复制仓库网址
[root@server2 ~]# helm repo list     ##列出repo仓库
[root@server2 ~]# helm search repo redis-ha    ##搜索redis-ha
NAME             	CHART VERSION	APP VERSION	DESCRIPTION                                       
dandydev/redis-ha	4.12.9       	6.0.7      	This Helm chart provides a highly available Red...
[root@server2 helm]# helm pull dandydev/redis-ha   ##从仓库拉取redis文件
[root@server2 helm]# tar zxf redis-ha-4.12.9.tgz    ##解压
[root@server2 helm]# cd redis-ha/    ##进入文件夹


##之前实验设置的nfs数据卷
[root@server2 redis-ha]# kubectl  get sc 
[root@server2 redis-ha]# kubectl get ns
[root@server2 redis-ha]# kubectl -n nfs-client-provisioner get all

## 修改配置文件
[root@server2 redis-ha]# vim values.yaml     ##关闭反亲和，是因为server2有污点
## hardAntiAffinity: false（有两个）
## 安装测试
[root@server2 redis-ha]# helm install redis-ha .    ##安装
[root@server2 redis-ha]# kubectl get pod
[root@server2 redis-ha]# kubectl get pv
[root@server2 redis-ha]# kubectl get pvc
[root@server2 redis-ha]# kubectl exec -it redis-ha-server-0 sh -n default  ##测试,并查看是否是master。可以删除一个查看会不会自动切换master

[root@server2 redis-ha]# kubectl delete pod redis-ha-server-0    ##删除之后查看master是否自动切换
[root@server2 redis-ha]# kubectl exec -it redis-ha-server-1 sh -n default   ##通过命令查看1和2两个主机的redis-cli下的info，查看master是否切换

Modify the configuration file

Installation and testing

3. Build a Helm Chart

3.1 Simple creation

[root@server2 helm]# helm  create mychart   ##创建
[root@server2 helm]# tree mychart/    ##查看
mychart/
├── charts
├── Chart.yaml  #编写mychart的应用描述信息
├── templates
│   ├── deployment.yaml
│   ├── _helpers.tpl
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── NOTES.txt
│   ├── serviceaccount.yaml
│   ├── service.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml    #编写应用部署信息

[root@server2 helm]# helm lint mychart/   ##检查依赖和模板配置是否正确
[root@server2 helm]# helm package mychart/    ##打包应用
[root@server2 helm]# ll mychart-0.1.0.tgz    
[root@server2 helm]# helm install demo mychart-0.1.0.tgz   ##安装 
[root@server2 helm]# kubectl get all   ##查看是否创建成功
[root@server2 helm]# kubectl get svc
[root@server2 helm]# helm  status demo     ##查看demo状态
[root@server2 helm]# helm list      ##列出仓库列表
[root@server2 helm]# helm show values mychart   ##展示helm细节

[root@server2 helm]# helm upgrade demo mychart-0.1.0.tgz --set replicaCount=2  ##更新，可以使用这种形式，也可以解压进去直接修改文件。更多使用方法参见官网

[root@server2 helm]# kubectl  get pod    ##

3.2 Establish a local chart warehouse

3.2.1 Create a new project in harbor warehouse

3.2.2 Install plug-in and upload

- 安装helm-push插件：
   $ helm plugin install https://github.com/chartmuseum/helm-push	//在线安装
  离线安装
	$ helm env	//获取插件目录
	$ mkdir ~/.local/share/helm/plugins/push
	$ tar zxf helm-push_0.8.1_linux_amd64.tar.gz -C ~/.local/share/helm/plugins/push
	$ helm push --help
	
	$ helm  repo list          ##列出仓库
	mychart  	http://172.25.0.11:30002/chartrepo/charts 

	$ helm push  mychart-0.1.0.tgz mychart -u admin -p Harbor12345    ##上传helm
	Pushing mychart-0.1.0.tgz to mychart...
	Done.
	$ helm  repo  update 		       ##更新仓库
	$ helm  search repo mychart   ##搜索

[root@server2 helm]# ll helm-push_0.9.0_linux_amd64.tar.gz   ##我使用的是离线包
-rw-r--r-- 1 root root 8943728 Dec  9 16:52 helm-push_0.9.0_linux_amd64.tar.gz
[root@server2 helm]# helm env   ##查看环境变量，确定插件位置
HELM_BIN="helm"
HELM_CACHE_HOME="/root/.cache/helm"
HELM_CONFIG_HOME="/root/.config/helm"
HELM_DATA_HOME="/root/.local/share/helm"
HELM_DEBUG="false"
HELM_KUBEAPISERVER=""
HELM_KUBEASGROUPS=""
HELM_KUBEASUSER=""
HELM_KUBECONTEXT=""
HELM_KUBETOKEN=""
HELM_MAX_HISTORY="10"
HELM_NAMESPACE="default"
HELM_PLUGINS="/root/.local/share/helm/plugins"    ##插件目录需要自己创建
HELM_REGISTRY_CONFIG="/root/.config/helm/registry.json"
HELM_REPOSITORY_CACHE="/root/.cache/helm/repository"
HELM_REPOSITORY_CONFIG="/root/.config/helm/repositories.yaml"

[root@server2 helm]# mkdir /root/.local/share/helm/plugins -p  ##创建目录
[root@server2 helm]# cd /root/.local/share/helm/plugins
[root@server2 plugins]# mkdir push   ##创建push目录
[root@server2 plugins]# cd 
[root@server2 ~]# cd helm/
[root@server2 helm]# tar zxf helm-push_0.9.0_linux_amd64.tar.gz -C /root/.local/share/helm/plugins/push/   ##解包到push目录

[root@server2 ~]# cd /etc/docker/certs.d/reg.westos.org/   ##复制证书
[root@server2 reg.westos.org]# cp ca.crt /etc/pki/ca-trust/source/anchors/
[root@server2 reg.westos.org]# update-ca-trust 


[root@server2 ~]# helm repo  add westos https://reg.westos.org/chartrepo/charts   ##添加仓库
[root@server2 ~]# helm repo list    ##列出仓库
NAME    	URL                                      
bitnami 	https://charts.bitnami.com/bitnami     
dandydev	https://dandydeveloper.github.io/charts
westos  	https://reg.westos.org/chartrepo/charts
[root@server2 ~]# cd helm/
[root@server2 helm]# helm push mychart-0.1.0.tgz westos --insecure -u admin -p westos  ##上传helm 
Pushing mychart-0.1.0.tgz to westos...
Done.
[root@server2 helm]# helm search repo mychart    ##搜索仓库
No results found
[root@server2 helm]# helm repo update     ##更新仓库
[root@server2 helm]# helm search repo mychart   ##搜索仓库
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                
westos/mychart	0.1.0        	v1         	A Helm chart for Kubernetes

[root@server2 helm]# helm uninstall  demo   ##卸载
[root@server2 helm]# helm show values westos/mychart   ##显示信息
 
[root@server2 helm]# helm install demo westos/mychart --set replicaCount=2  ##建立俩个pod并测试
[root@server2 helm]# kubectl get svc 
NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
demo-mychart   ClusterIP   10.108.247.81   <none>        80/TCP    88s
kubernetes     ClusterIP   10.96.0.1       <none>        443/TCP   13d
[root@server2 helm]# curl 10.108.247.81
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@server2 helm]# curl 10.108.247.81/hostname.html
demo-mychart-8568b558d7-69b8k
[root@server2 helm]# curl 10.108.247.81/hostname.html
demo-mychart-8568b558d7-dvcvx

3.2.3 Update and rollback (version)

[root@server2 mychart]# vim values.yaml 
[root@server2 mychart]# vim values.yaml    ##设置image的tag为v2
[root@server2 mychart]# vim Chart.yaml 
[root@server2 mychart]# cd ..
[root@server2 helm]# helm package mychart
Successfully packaged chart and saved it to: /root/helm/mychart-0.2.0.tgz 
[root@server2 helm]# helm push mychart-0.2.0.tgz westos --insecure -u admin -p westos
Pushing mychart-0.2.0.tgz to westos...
Done.
[root@server2 helm]# helm repo update 
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "westos" chart repository
...Unable to get an update from the "dandydev" chart repository (https://dandydeveloper.github.io/charts):
	Get "https://dandydeveloper.github.io/charts/index.yaml": read tcp 172.25.13.2:55026->185.199.111.153:443: read: connection reset by peer
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

##更新
[root@server2 mychart]# vim values.yaml 
[root@server2 mychart]# helm upgrade demo westos/mychart   #更新
[root@server2 helm]# helm search repo mychart -l
NAME          	CHART VERSION	APP VERSION	DESCRIPTION                
westos/mychart	0.2.0        	v2         	A Helm chart for Kubernetes
westos/mychart	0.1.0        	v1         	A Helm chart for Kubernetes

[root@server2 helm]# kubectl -n ingress-nginx get all     ##查看分配的ip
[root@westos Desktop]# vim /etc/hosts     ##真机作解析
[root@westos Desktop]# cat /etc/hosts | grep demo.westos.org
172.25.13.100 demo.westos.org
[root@westos Desktop]# curl demo.westos.org    ##真机访问


##回滚
[root@server2 mychart]# helm history demo 
REVISION	UPDATED                 	STATUS    	CHART        APP VERSION	DESCRIPTION     
1       	Fri Mar  5 10:28:09 2021	superseded	mychart-0.1.0v1         	Install complete
2       	Fri Mar  5 10:28:54 2021	superseded	mychart-0.1.0v1         	Upgrade complete
3       	Fri Mar  5 10:37:30 2021	superseded	mychart-0.2.0v2         	Upgrade complete
4       	Fri Mar  5 10:47:13 2021	deployed  	mychart-0.2.0v2         	Upgrade complete
[root@server2 mychart]# helm rollback demo  1
[root@server2 mychart]# kubectl  get ingress    ##1版本是没有ingress的
No resources found in default namespace.
[root@server2 mychart]# kubectl  get pod

4. Helm deploys metrics-server application

## 1. 拉取镜像chart仓库和app镜像
[root@server1 harbor]# docker images | grep metrics-server   ##版本对应value.yaml文件
bitnami/metrics-server                                     0.4.2-debian-10-r0               99c9b9952763   3 weeks ago     171MB

[root@server2 ~]# helm search repo metrics-server  ##查看对应的chart版本和app版本，然后下载对应的app版本的镜像
[root@server2 helm]# helm pull bitnami/metrics-server
[root@server2 helm]# tar zxf metrics-server-5.6.0.tgz 

## 2. 修改配置文件value.yaml
[root@server2 metrics-server]# vim values.yaml  ##修改内容看下面截图

## 3. 创建namespace并安装
[root@server2 metric-server]# kubectl create namespace metrics-server
[root@server2 metrics-server]# helm install metrics-server . -n metrics-server 
## 此时pod起不来，原因是无法解析节点

## 4. 修改corndns
[root@server2 metrics-server]# kubectl -n kube-system edit cm coredns   ##修改内容如下图
[root@server2 metrics-server]# kubectl -n metrics-server delete pod metrics-server-777f7bd69b-4xcd6   ##删除节点，会重建，相当于更新
[root@server2 metrics-server]# kubectl -n kube-system get pod | grep coredns | awk '{system("kubectl -n kube-system delete pod "$1"")}'    ##删除重建，相当于更新一下coredns节点
## 5. 查看节点是否运行

1. Pull the mirror image of the corresponding app version

2. Modify the configuration file

3. Create and install

4. Modify corndns

5. Check if the node is running

5. Helm provides web UI interface management

- 部署kubeapps应用，为Helm提供web UI界面管理：
	$ helm repo add bitnami https://charts.bitnami.com/bitnami
	$ helm pull bitnami/kubeapps
	$ vim values.yaml
		global:
		  imageRegistry: reg.westos.org		
		useHelm3: true
		ingress:
		  enabled: true
		  hostname: kubeapps.westos.org
	
	$ kubectl create namespace kubeapps
	$ helm install kubeapps -n kubeapps .
	
	$ kubectl create serviceaccount kubeapps-operator -n kubeapps
	$ kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=kubeapps:kubeapps-operator

## 5.1 拉取Kubeapps下value.yaml中的镜像文件
#直接看下图
## 5.2 真机配置解析
[root@westos Desktop]# vim /etc/hosts
[root@westos Desktop]# cat /etc/hosts | grep kubeapps.westos.org
172.25.13.100 demo.westos.org kubeapps.westos.org
## 5.3 修改value配置文件并运行
[root@server2 helm]# helm  repo add bitnami https://charts.bitnami.com/bitnami  ##添加bitnami仓库
[root@server2 helm]# helm pull bitnami/kubeapps  ##拉取kubeapps文件
[root@server2 kubeapps]# vim values.yaml 
[root@server2 kubeapps]# kubectl create namespace kubeapps    ##创建命名空间
[root@server2 kubeapps]# helm install kubeapps -n kubeapps .  ##安装
[root@server2 kubeapps]# kubectl create serviceaccount kubeapps-operator -n kubeapps  ##创建sa
[root@server2 kubeapps]# kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=kubeapps:kubeapps-operator    ##

[root@server2 kubeapps]# kubectl -n kubeapps get pod
[root@server2 kubeapps]# kubectl -n kubeapps describe sa kubeapps-operator    ##查看secrets
Name:                kubeapps-operator
Namespace:           kubeapps
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
Mountable secrets:   kubeapps-operator-token-g5kw6
Tokens:              kubeapps-operator-token-g5kw6
Events:              <none>
[root@server2 kubeapps]# kubectl -n kubeapps describe secrets kubeapps-operator-token-g5kw6   ##查看token

##5.4 网页测试

5.1 Pull the image file in value.yaml under Kubeapps

5.2 Real device configuration analysis

5.3 Modify the value configuration file

5.4 Web page test

5.5 Add local warehouse to kubeapps