avi与原生K8S集成，完成POD网络对外提供服务的功能替代。
本文记录安装实现过程。
环境组件：
| 名称 | 版本 | 备注 |
|---|---|---|
| vCenter | 7.0u1 | |
| AVI Controller | 20.1.3 | |
| AKO | 1.3.1 | |
| Kubernestes Node | 1.20.2 |
拓扑图与IP地址规划
简单的把AVI Controller，SE，VC，Node都放着一个网段。
| 名称 | IP地址 | 备注 |
|---|---|---|
| vCenter | 10.105.130.18/26 | |
| AVI Controller | 10.105.130.55/26 | |
| K8S Nodes | 10.105.130.30-32/26 | |
| AVI SE | 10.105.130.41-50/26 |
实验步骤
1.部署 Avi Controller ，通常通过 VMware vCenter 部署
2.通过 Avi Controller 完成 Cloud 相关配置，确保 SE 可以正常配置和工作
3.通过 Helm 添加 repo（或者离线获取 repo），使用 helm install 设置参数并安装 AKO （或者修改离线 repo 中的 values.yaml 然后再安装）
4.AKO 正确部署到 k8s 集群中
5.AKO 自动与 Avi Controller 建立连接
6.可以开始部署应用并创建服务了！
部署 Avi Controller ，通常通过 VMware vCenter 部署
这部分内容参考：AVI vCenter Cloud配置
Avi Controller 完成 Cloud 相关配置
•通过 vCenter 导入 ova 部署，根据向导完成虚拟机资源、IP 地址等配置
•通过浏览器登陆 Avi 界面进行初始化，需要进行密码、域名、NTP 等配置
IPAM需要把10.105.130.0/26段包括，并划出Se可以使用的地址段
内部的DNS Service沿用前面实验中的。
通过 Helm 添加 repo并安装AKO
Helm安装，最新为v3.5
官方安装文档
这里使用：
From Apt (Debian/Ubuntu)
Members of the Helm community have contributed a Helm package for Apt. This package is generally up to date.
curl https://baltocdn.com/helm/signing.asc | sudo apt-key add -
sudo apt-get install apt-transport-https --yes
echo "deb https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
完成以后发现装在了 /usr/sbin/helm
移动到默认位置：mv /usr/sbin/helm /usr/local/bin/helm
helm repo add ako https://avinetworks.github.io/avi-helm-charts/charts/stable/ako/
root@ubuntu-master:~# helm search repo
NAME CHART VERSION APP VERSION DESCRIPTION
ako/ako 1.3.1 1.3.1 A helm chart for Avi Kubernetes Operator
ako/ako-operator 1.3.1 1.3.1 A Helm chart for Kubernetes AKO Operator
安装AKO要注意参数：
一开始使用required的参数：
helm install ako ako/ako --version 1.3.1 \
--set ControllerSettings.controllerHost=10.105.130.55 --set avicredentials.username=admin \
--set avicredentials.password=VMware1! --set ControllerSettings.controllerVersion="20.1.3" \
--set AKOSettings.clusterName=ako-cluster-001 --set NetworkSettings.subnetIP=10.105.130.0 \
--set NetworkSettings.subnetPrefix=26 --set NetworkSettings.networkName=DPortGroup-MGMT-01a \
--set AKOSettings.cniPlugin=calico --set AKOSettings.disableStaticRouteSync=false \
--set L7Settings.shardVSSize=SMALL --namespace=avi-system
发现出错：
root@ubuntu-master:~# kubectl logs -f ako-0 -n avi-system
2021-01-20T03:12:07.028Z INFO api/api.go:52 Setting route for GET /api/status
2021-01-20T03:12:07.028Z INFO ako-main/main.go:61 AKO is running with version: v1.3.1
2021-01-20T03:12:07.028Z INFO api/api.go:110 Starting API server at :8080
2021-01-20T03:12:07.028Z INFO ako-main/main.go:67 We are running inside kubernetes cluster. Won't use kubeconfig files.
2021-01-20T03:12:07.126Z INFO utils/ingress.go:39 networking.k8s.io/v1/IngressClass enabled on cluster
2021-01-20T03:12:07.126Z INFO utils/utils.go:166 Initializing configmap informer in avi-system
2021-01-20T03:12:07.683Z INFO cache/avi_ctrl_clients.go:72 Setting the client version to 20.1.3
2021-01-20T03:12:07.683Z INFO cache/avi_ctrl_clients.go:72 Setting the client version to 20.1.3
2021-01-20T03:12:07.902Z INFO cache/controller_obj_cache.go:2641 Setting cloud vType: CLOUD_NONE
2021-01-20T03:12:07.902Z ERROR cache/controller_obj_cache.go:2646 Cloud does not have a ipam_provider_ref configured
2021-01-20T03:12:07.902Z INFO lib/lib.go:70 Setting AKOUser: ako-ako-cluster-001 for Avi Objects
2021-01-20T03:12:07.911Z WARN cache/controller_obj_cache.go:2474 Invalid input detected, AKO will be rebooted to retry
2021-01-20T03:12:07.911Z INFO api/api.go:68 Shutting down the API server
2021-01-20T03:12:07.911Z INFO api/api.go:113 API server shutdown: http: Server closed
2021-01-20T03:12:08.411Z WARN cache/controller_obj_cache.go:2477 Invalid input detected, sync will be disabled.
2021-01-20T03:12:08.411Z ERROR ako-main/main.go:134 Handleconfigmap error during reboot, shutting down AKO
2021-01-20T03:12:07.902Z ERROR cache/controller_obj_cache.go:2646 Cloud does not have a ipam_provider_ref configure
但是在AVI Controller上已经建好了IPAM，此处说找不到？
仔细参考log发现：
2021-01-20T03:12:07.902Z INFO cache/controller_obj_cache.go:2641 Setting cloud vType: CLOUD_NONE
而在设置里面Cloud为：avi-vcsa-01a
增加一条参数指定Cloud：
helm install ako ako/ako --version 1.3.1 \
--set ControllerSettings.controllerHost=10.105.130.55 --set avicredentials.username=admin \
--set avicredentials.password=VMware1! --set ControllerSettings.controllerVersion="20.1.3" \
--set AKOSettings.clusterName=ako-cluster-001 --set NetworkSettings.subnetIP=10.105.130.0 \
--set NetworkSettings.subnetPrefix=26 --set NetworkSettings.networkName=DPortGroup-MGMT-01a \
*--set ControllerSettings.cloudName=avi-vcsa-01a \*
--set AKOSettings.cniPlugin=calico --set AKOSettings.disableStaticRouteSync=false \
--set L7Settings.shardVSSize=SMALL --namespace=avi-system
执行后成功：
root@ubuntu-master:~# helm delete ako -n avi-system
release "ako" uninstalled
root@ubuntu-master:~# helm install ako ako/ako --version 1.3.1 \
> --set ControllerSettings.controllerHost=10.105.130.55 --set avicredentials.username=admin \
> --set avicredentials.password=VMware1! --set ControllerSettings.controllerVersion="20.1.3" \
> --set AKOSettings.clusterName=ako-cluster-001 --set NetworkSettings.subnetIP=10.105.130.0 \
> --set NetworkSettings.subnetPrefix=26 --set NetworkSettings.networkName=DPortGroup-MGMT-01a \
> --set ControllerSettings.cloudName=avi-vcsa-01a \
> --set AKOSettings.cniPlugin=calico --set AKOSettings.disableStaticRouteSync=false \
> --set L7Settings.shardVSSize=SMALL --namespace=avi-system
NAME: ako
LAST DEPLOYED: Wed Jan 20 11:29:56 2021
NAMESPACE: avi-system
STATUS: deployed
REVISION: 1
root@ubuntu-master:~# kubectl logs -f ako-0 -n avi-system
Error from server (NotFound): pods "ako-0" not found
root@ubuntu-master:~# kubectl get po -n avi-system
NAME READY STATUS RESTARTS AGE
ako-0 1/1 Running 0 13s
root@ubuntu-master:~# kubectl logs -f ako-0 -n avi-system
2021-01-20T03:30:22.401Z INFO api/api.go:52 Setting route for GET /api/status
2021-01-20T03:30:22.401Z INFO ako-main/main.go:61 AKO is running with version: v1.3.1
2021-01-20T03:30:22.401Z INFO ako-main/main.go:67 We are running inside kubernetes cluster. Won't use kubeconfig files.
2021-01-20T03:30:22.402Z INFO api/api.go:110 Starting API server at :8080
2021-01-20T03:30:22.461Z INFO utils/ingress.go:39 networking.k8s.io/v1/IngressClass enabled on cluster
2021-01-20T03:30:22.461Z INFO utils/utils.go:166 Initializing configmap informer in avi-system
2021-01-20T03:30:22.963Z INFO cache/avi_ctrl_clients.go:72 Setting the client version to 20.1.3
2021-01-20T03:30:22.963Z INFO cache/avi_ctrl_clients.go:72 Setting the client version to 20.1.3
2021-01-20T03:30:23.099Z INFO cache/controller_obj_cache.go:2641 Setting cloud vType: CLOUD_VCENTER
2021-01-20T03:30:23.099Z INFO lib/lib.go:70 Setting AKOUser: ako-ako-cluster-001 for Avi Objects
......
查看部署情况：
root@ubuntu-master:~# helm list -n avi-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
ako avi-system 1 2021-01-20 11:29:56.831070472 +0800 CST deployed ako-1.3.1 1.3.1
root@ubuntu-master:~# kubectl get po -n avi-system -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ako-0 1/1 Running 0 26h 172.38.11.198 ubuntu-worker01 <none> <none>
验证
建一个简单应用试试：
apiVersion: v1
kind: Service
metadata:
name: hello-kubernetes
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
selector:
app: hello-kubernetes
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-kubernetes
spec:
replicas: 3
selector:
matchLabels:
app: hello-kubernetes
template:
metadata:
labels:
app: hello-kubernetes
spec:
containers:
- name: hello-kubernetes
image: paulbouwer/hello-kubernetes:1.5
ports:
- containerPort: 8080
env:
- name: MESSAGE
value: I just deployed Web Service via AVI for pod Cluster!!
执行后可以查看：
root@ubuntu-master:~# kubectl apply -f hello-depolyment.yaml
service/hello-kubernetes created
deployment.apps/hello-kubernetes created
root@ubuntu-master:~# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-kubernetes LoadBalancer 10.100.200.216 10.105.130.43 80:32243/TCP 5s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
我们可以看到这个服务对外的地址：10.105.130.43，通过浏览器验证：
在AVI的控制面板上：
可以查看到hello服务的dns地址：hello-kubernetes.default.avi.vmlab.local
通过浏览器验证：
通过命令修改hello-deployment的pod数量
root@ubuntu-master:~# kubectl scale deployment --replicas=6 hello-kubernetes
deployment.apps/hello-kubernetes scaled
root@ubuntu-master:~# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-kubernetes-8f5997554-5bgxh 1/1 Running 0 27m 172.38.11.210 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-5tzh5 1/1 Running 0 27m 172.38.184.206 ubuntu-worker02 <none> <none>
hello-kubernetes-8f5997554-96hzx 1/1 Running 0 27m 172.38.184.205 ubuntu-worker02 <none> <none>
hello-kubernetes-8f5997554-d8442 1/1 Running 0 9s 172.38.11.211 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-h76hr 1/1 Running 0 9s 172.38.11.212 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-tnwr6 1/1 Running 0 9s 172.38.184.207 ubuntu-worker02 <none> <none>
再次查看AVI控制界面：
AVI对LB的负载进行了自动扩容。
再次对hello服务进行缩容：
root@ubuntu-master:~# kubectl scale deployment --replicas=4 hello-kubernetes
deployment.apps/hello-kubernetes scaled
root@ubuntu-master:~# kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-kubernetes-8f5997554-5bgxh 1/1 Running 0 32m 172.38.11.210 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-5tzh5 1/1 Running 0 32m 172.38.184.206 ubuntu-worker02 <none> <none>
hello-kubernetes-8f5997554-96hzx 1/1 Running 0 32m 172.38.184.205 ubuntu-worker02 <none> <none>
hello-kubernetes-8f5997554-d8442 0/1 Terminating 0 5m8s 172.38.11.211 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-h76hr 1/1 Running 0 5m8s 172.38.11.212 ubuntu-worker01 <none> <none>
hello-kubernetes-8f5997554-tnwr6 0/1 Terminating 0 5m8s <none> ubuntu-worker02 <none> <none>
结论：AVI和K8S的集成可用，能够实现自动Ingress LB，yaml或命令方式简单，支持自动扩缩容。
本文选择了AVI/AKO与Kubernetes的简单集成实现，更多方式和参考如下：
avi-helm-charts
Avi Kubernetes Operator
以上。