I began to think that as long as the installation prometheus-operator on the line.
git clone https://github.com/coreos/prometheus-operator.git
cd prometheus-operator
sed 's/namespace: default/namespace: monitoring/g' bundle.yaml | kubectl apply -f -
After installing found only one prometheus-operator pod.
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-operator-99dccdc56-qq5lm 1/1 Running 0 20m
But then I noticed that really want to install Kube-Prometheus .
https://github.com/coreos/kube-prometheus.git
cd kube-prometheus
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl create -f manifests/
kube-prometheus prometheus contains all the components used in the monitor, of course, also included prometheus-operator.
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 112m
grafana-58dc7468d7-pv256 1/1 Running 0 112m
kube-state-metrics-769f4fd4d5-4hfpk 3/3 Running 0 112m
node-exporter-4grdx 2/2 Running 0 112m
prometheus-adapter-5cd5798d96-6zq6f 1/1 Running 0 112m
prometheus-k8s-0 3/3 Running 1 112m
prometheus-operator-99dccdc56-vqq7m 1/1 Running 0 112m
Before installing prometheus, run kubectl top command error.
$ kubectl top node
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
This is because not installed metrics-server, there is such a description in the text of the readme kube-prometheus:
The kube-prometheus stack includes a resource metrics API server, so the metrics-server addon is not necessary. Ensure the metrics-server addon is disabled on minikube.
So you know kube-prometheus contains resource metrics API server, so after installing kube-prometheus kubectl top node can be properly executed.
ubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master0 1715m 85% 1001Mi 26%
k8s-master1 347m 17% 972Mi 25%
k8s-master2 286m 14% 1016Mi 26%
k8s-node4 373m 9% 1550Mi 19%
After installing prometheus exposed by kubectl port-forward command port, you can access Prometheus, Grafana, AlertManager console through the browser separately, the effect of the console see Kubernetes Monitoring with Prometheus .
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090 --address 10.0.1.81
kubectl port-forward $(kubectl get pods --selector=app=grafana -n monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring 3000 --address 10.0.1.81
kubectl port-forward -n monitoring alertmanager-main-0 9093 --address 10.0.1.81
Note: The above 10.0.1.81 into your server IP address, if you do not specify --address, accessible only via localhost.