Teach you how to use Prometheus to monitor JVM

Overview

When your Java business is containerized on K8S, how to monitor it? The Prometheus community has developed JMX Exporter to export the monitoring indicators of the JVM so that Prometheus can be used to collect monitoring data. This article will introduce how to use Prometheus and JMX Exporter to monitor the JVM of your Java application.

What is JMX Exporter?

JMX Exporter uses Java's JMX mechanism to read some monitoring data during JVM runtime, and then converts it into the metrics format recognized by Prometheus so that Prometheus can monitor and collect it.

So, what is JMX? Its full name Java Management Extensionsis: . As the name implies, it is an extended framework for managing Java. JMX Exporter is based on this framework to read the runtime state of the JVM.

How to use JMX Exporter to expose JVM monitoring metrics?

The following describes how to expose the JVM monitoring indicators of Java applications through JMX Exporter.

Two usages of JMX Exporter

JMX-Exporter provides two usages:

1. Start an independent process. Specify parameters when JVM starts, and expose the RMI interface of JMX. JMX-Exporter calls RMI to obtain JVM runtime state data, converts it to Prometheus metrics format, and exposes the port for Prometheus to collect.
2. JVM starts in-process. Specify the parameters when the JVM starts, run the JMX-Exporter jar package in the form of javaagent, read the JVM runtime status data in the process, convert it to the Prometheus metrics format, and expose the port for Prometheus to collect.

Officials do not recommend the first method. On the one hand, the configuration is complicated, and on the other hand, because it requires a separate process, and the monitoring of this process itself has become a new problem, this article focuses on the second usage and how to use K8S Use JMX Exporter to expose JVM monitoring indicators in the environment.

Packaged image

Using the second usage, you need to specify the JMX Exporter jar file and configuration file when starting the JVM. The jar package is a binary file, and it is not easy to mount it through configmap. We hardly need to modify the configuration file, so it is recommended to directly package the JMX Exporter jar package and configuration file into the business container image.

First prepare a directory for making a mirror, and put the JMX Exporter configuration file prometheus-jmx-config.yaml:

---
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false

Note : For more configuration items, please refer to the official documentation.

Then prepare the jar package file. You can find the latest jar package download address on the Github page of jmx_exporter and download it to the current directory:

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar

Then prepare the Dockerfile, here is tomcat as an example:

FROM tomcat:jdk8-openjdk-slim
ADD prometheus-jmx-config.yaml /prometheus-jmx-config.yaml
ADD jmx_prometheus_javaagent-0.13.0.jar /jmx_prometheus_javaagent-0.13.0.jar

Finally compile the image:

docker build . -t ccr.ccs.tencentyun.com/imroc/tomcat:jdk8

Get it done! If you want to be simpler, you can use docker multi-stage construction to save the steps of manually downloading the jar package. Dockerfile example:

FROM ubuntu:16.04 as jar
WORKDIR /
RUN apt-get update -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y wget
RUN wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.13.0/jmx_prometheus_javaagent-0.13.0.jar

FROM tomcat:jdk8-openjdk-slim
ADD prometheus-jmx-config.yaml /prometheus-jmx-config.yaml
COPY --from=jar /jmx_prometheus_javaagent-0.13.0.jar /jmx_prometheus_javaagent-0.13.0.jar

Deploy Java applications

With the packaged image, the next step is to look at how to deploy the application to K8S. The key point is how to modify the JVM startup parameters to load the JMX Exporter at startup. JVM reads at startup JAVA_OPTSthe environment variable, as an additional start-up parameters, so we can use to increase the deployment of what this environment variable, the example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tomcat
  template:
    metadata:
      labels:
        app: tomcat
    spec:
      containers:
      - name: tomcat
        image: ccr.ccs.tencentyun.com/imroc/tomcat:jdk8
        env:
        - name: JAVA_OPTS
          value: "-javaagent:/jmx_prometheus_javaagent-0.13.0.jar=8088:/prometheus-jmx-config.yaml"

---
apiVersion: v1
kind: Service
metadata:
  name: tomcat
  labels:
    app: tomcat
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    name: http
  - port: 8088
    protocol: TCP
    name: jmx-metrics
  selector:
    app: tomcat

• Start parameter format: -javaagent:<jar>=<port>:<config>.
• Here, port 8088 is used to expose the monitoring indicators of JVM, which can be changed by yourself.

Add Prometheus monitoring configuration

The monitoring indicators of the JVM are exposed. Now let's configure Prometheus so that the monitoring data is collected. Configuration example:

    - job_name: tomcat
      scrape_interval: 5s
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - default
      relabel_configs:
      - action: keep
        source_labels:
        - __meta_kubernetes_service_label_app
        regex: tomcat
      - action: keep
        source_labels:
        - __meta_kubernetes_endpoint_port_name
        regex: jmx-metrics

If prometheus-operator is installed, you can also configure Prometheus by creating a CRD object of ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: tomcat
  namespace: default
  labels:
    app: tomcat
spec:
  endpoints:
  - port: jmx-metrics
    interval: 5s
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app: tomcat

Add Grafana monitoring panel

After collecting the data, let's see how to display the data. If you are familiar with Prometheus and Grafana, you can design the panels you need according to the indicators. The community also provides ready-made panels, but they are a bit old, and some views may not be displayed. It is https://grafana.com/grafana/dashboards/8563 , which can be imported directly. Panel rendering:

Reference

• JMX Exporter project address: https://github.com/prometheus/jmx_exporter
• JVM monitoring panel: https://grafana.com/grafana/dashboards/8563