Heapster是容器集群监控和性能分析工具，HPA、Dashborad、Kubectl top都依赖于heapster收集的数据。
但是Heapster从kubernetes 1.8以后已经被遗弃了...... 被metrics-server所替代......
kubernetes 1.11 中部署Heapster 1.5.4版本的过程
Heapster 部署yaml文件
apiVersion: v1 kind: ServiceAccount metadata: name: heapster namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: heapster roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:heapster subjects: - kind: ServiceAccount name: heapster namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: heapster namespace: kube-system spec: replicas: 1 template: metadata: labels: task: monitoring k8s-app: heapster spec: serviceAccountName: heapster containers: - name: heapster # image: k8s.gcr.io/heapster-amd64:v1.5.4 将默认google的官方镜像替换为阿里云镜像，否则你懂得 image: registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4 command: - /heapster - --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true --- apiVersion: v1 kind: Service metadata: labels: task: monitoring # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons) # If you are NOT using this as an addon, you should comment out this line. kubernetes.io/cluster-service: 'true' kubernetes.io/name: Heapster name: heapster namespace: kube-system spec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster
heapster 启动参数说明：
inClusterConfig- Use kube config in service accounts associated with Heapster's namespace. (default: true)kubeletPort- kubelet port to use (default:10255)kubeletHttps- whether to use https to connect to kubelets (default:false)insecure- whether to trust Kubernetes certificates (default:false)auth- client auth file to use. Set auth if the service accounts are not usable.useServiceAccount- whether to use the service account token if one is mounted at/var/run/secrets/kubernetes.io/serviceaccount/token(default:false)
使用： kubectl apply -f heapster.yaml 部署
部署后查看log日志，发现一直提示“403 Forbidden”, response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)"
[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-855k7 I0912 14:36:48.175115 1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true I0912 14:36:48.175174 1 heapster.go:79] Heapster version v1.5.4 I0912 14:36:48.175366 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1 I0912 14:36:48.175393 1 configs.go:62] Using kubelet port 10250 I0912 14:36:48.185828 1 heapster.go:202] Starting with Metric Sink I0912 14:36:48.199515 1 heapster.go:112] Starting heapster on port 8082 E0912 14:37:05.000327 1 kubelet.go:288] node node01 is not ready E0912 14:38:05.016044 1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)" W0912 14:38:25.000643 1 manager.go:152] Failed to get all responses in time (got 0/1) E0912 14:39:05.008534 1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)" W0912 14:39:25.000674 1 manager.go:152] Failed to get all responses in time (got 0/1) E0912 14:40:05.009955 1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)" W0912 14:40:25.001231 1 manager.go:152] Failed to get all responses in time (got 0/1) E0912 14:41:05.017198 1 manager.go:101] Error in scraping containers from kubelet:172.16.65.181:10250: failed to get all container stats from Kubelet URL "https://172.16.65.181:10250/stats/container/": request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=create, resource=nodes, subresource=stats)" W0912 14:41:25.000846 1 manager.go:152] Failed to get all responses in time (got 0/1)
查看ClusterRole: system:heapster的权限，发现的确没有针对Resource: nodes/stats 的create权限
[root@node01 heapster-yaml]# kubectl describe clusterrole system:heapster Name: system:heapster Labels: kubernetes.io/bootstrapping=rbac-defaults Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"lab... rbac.authorization.kubernetes.io/autoupdate=true PolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- events [] [] [get list watch] namespaces [] [] [get list watch] nodes [] [] [get list watch] pods [] [] [get list watch] deployments.extensions [] [] [get list watch]
修改ClusterRole: system:heapster的权限:
1. 查看system:heapster yaml格式, 保存为 heapster-clusterrole.yaml
yaml
[root@node01 heapster-yaml]# kubectl get clusterrole system:heapster -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{"rbac.authorization.kubernetes.io/autoupdate":"true"},"labels":{"kubernetes.io/bootstrapping":"rbac-defaults"},"name":"system:heapster","namespace":""},"rules":[{"apiGroups":[""],"resources":["events","namespaces","nodes","pods","nodes/stats"],"verbs":["create","get","list","watch"]},{"apiGroups":["extensions"],"resources":["deployments"],"verbs":["get","list","watch"]}]} rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: 2018-08-26T02:26:14Z labels: kubernetes.io/bootstrapping: rbac-defaults name: system:heapster resourceVersion: "139000" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Aheapster uid: 67ef3689-a8d7-11e8-a891-000c29b52823 rules: - apiGroups: - "" resources: - events - namespaces - nodes - pods verbs: - get - list - watch - apiGroups: - extensions resources: - deployments verbs: - get - list - watch
2. 添加Resource: nodes/stats的create权限，并执行 kubectl apply -f heapster-clusterrole.yaml
yaml
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" labels: kubernetes.io/bootstrapping: rbac-defaults name: system:heapster rules: - apiGroups: - "" resources: - events - namespaces - nodes - pods - nodes/stats verbs: - create - get - list - watch - apiGroups: - extensions resources: - deployments verbs: - get - list - watch
3. 删除heapster重新部署
kubectl delete -f heapster.yaml
kubectl apply -f heapster.yaml
4. 重新部署后，查看log没有任何报错了
[root@node01 heapster-yaml]# kubectl logs --namespace=kube-system heapster-868d5cd7f-8zgxq I0912 15:02:38.926068 1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true I0912 15:02:38.926128 1 heapster.go:79] Heapster version v1.5.4 I0912 15:02:38.926565 1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1 I0912 15:02:38.926647 1 configs.go:62] Using kubelet port 10250 I0912 15:02:38.938333 1 heapster.go:202] Starting with Metric Sink I0912 15:02:38.949215 1 heapster.go:112] Starting heapster on port 8082
5. 功能测试
部署前：
[root@node01 heapster-yaml]# kubectl top pod Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
部署后：
[root@node01 tomcat-mysql-yaml]# kubectl top pod NAME CPU(cores) MEMORY(bytes) lxcfs-j8bpd 0m 4Mi mysql-85c85c5668-fr6j7 0m 458Mi tomcat-78c9778858-424gc 1m 93Mi tomcat-78c9778858-42xz4 1m 97Mi tomcat-78c9778858-6t7cd 1m 93Mi
相关链接：
https://github.com/kubernetes/heapster
https://github.com/kubernetes-incubator/metrics-server
https://kubernetes.io/docs/tasks/debug-application-cluster/core-metrics-pipeline/