JVM monitoring under OpenShift

Last year wrote an article based on jmx monitoring, this time implemented on Openshift, really found a lot of changes. The main issues in focus

1. prometheus jmx exporter improved, eliminating the need to run a separate process, the input data does not need to influxdb,

But directly provides prometheus data. Link

https://github.com/prometheus/jmx_exporter

2. OpenShift using the prometheus Operator architecture with a cluster of monitoring indicators, how to integrate monitoring data of customers on this basis.

Next we look at the specific implementation process.

1. Modify the application for mirroring

Download agent's jmx_prometheus_javaagent-0.12.0.jarpackage, specific links

https://github.com/prometheus/jmx_exporter

To tomcat, for example, build a mirror

[root@master jmx]# ls
catalina.sh  config.yaml  Dockerfile  jmx_prometheus_javaagent-0.12.0.jar

Dockerfile

[root@master jmx]# cat Dockerfile 
FROM registry.example.com/tomcat:8-slim 

COPY *.* /usr/local/tomcat/bin/

CMD ["catalina.sh", "run"]

catalina.sh copied from the mirror out, place the following modifications

CLASSPATH=/usr/local/tomcat/bin

JAVA_OPTS="-javaagent:/usr/local/tomcat/bin/jmx_prometheus_javaagent-0.12.0.jar=8180:/usr/local/tomcat/bin/config.yaml $JAVA_OPTS $JSSE_OPTS"

Because the default port of 8080 and start conflicts tomcat, so changed to 8180

config.yaml, the simplest configuration, what information is acquired

---   
lowercaseOutputLabelNames: true
lowercaseOutputName: true

Then build deployment

docker build -t registry.example.com/tomcatjmx:8-slim .

docker push registry.example.com/tomcatjmx:8-slim

oc import-image tomcatjmx:8-slim --from=registry.example.com/tomcatjmx:8-slim --confirm --insecure=true

After completion of the saw at the image myproject have tomcatjmx: 8-slim

8180 open ports in the service, modify yaml file, add port 8180

    - name: 8180-tcp
      port: 8180
      protocol: TCP
      targetPort: 8180

And then apply the image is nearly complete.

2.prometheus add target

OpenShift of prometheus using Operator to deploy, based servicemonitor model for service monitoring.

Servicemonitor built in OpenShift environment

[root@master jmx]# oc -n openshift-monitoring get servicemonitor
NAME                          AGE
alertmanager                  175d
cluster-monitoring-operator   175d
etcd                          6d
kube-apiserver                175d
kube-controllers              175d
kube-state-metrics            175d
kubelet                       175d
node-exporter                 175d
prometheus                    175d
prometheus-operator           175d

service monitor通过namespace确定工作范围，同时基于Label对服务(Service)进行监控

我们如果需要对自己的服务进行监控，就需要建立一个客户化的Service monitor

[root@master ~]# cat myservicemonitor.yaml 
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tomcatmonitor 
  labels:
    k8s-app: tomcatmonitor
  namespace: openshift-monitoring
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      app: tomcatjmx
  endpoints:
    - interval: 30s
      path: /metrics
      port: 8180-tcp

比较核心的地方是selector, 决定对哪个服务进行监控，另外就是endpoint,针对哪个port获取指标。

创建后可以通过下面命令看到

[root@master ~]# oc -n openshift-monitoring get servicemonitor
NAME                          AGE
alertmanager                  175d
cluster-monitoring-operator   175d
etcd                          6d
kube-apiserver                175d
kube-controllers              175d
kube-state-metrics            175d
kubelet                       175d
node-exporter                 175d
prometheus                    175d
prometheus-operator           175d
tomcatmonitor                 1h

但这时候在prometheus的target上是看不到我们tomcatmonitor的，因为缺少了授权工作，添加RBAC的授权

oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s

oc adm policy add-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s -n myproject

完成后，在prometheus target界面上看到tomcatjmx服务已经启动