gorilla/mux framework (rk-boot): add Prometheus monitoring middleware

introduce

Through a complete example, add Prometheus monitoring middleware to microservices based on gorilla/mux framework.

What is a Prometheus monitoring interceptor/middleware?

The monitoring interceptor records Prometheus Metrics for each API request.

We will use rk-boot to start the gorilla/mux microservice .

Please visit the following address for the full tutorial: https://github.com/rookie-ninja/rk-mux

Install

go get github.com/rookie-ninja/rk-boot/mux

quick start

1. Create boot.yaml

The boot.yaml file describes the original information of Mux framework startup, rk-boot starts GoFrame by reading boot.yaml.

To verify, we enabled the following options:

• commonService : commonService contains a series of common APIs. Details
• prom : Prometheus (Prometheus) client.
• prometheus middleware : Start the prometheus middleware.

---
mux:
  - name: greeter                     # Required
    port: 8080                        # Required
    enabled: true                     # Required
    prom:
      enabled: true                   # Optional, default: false
    commonService:
      enabled: true                   # Optional, default: false
    interceptors:
      metricsProm:
        enabled: true                 # Optional, default: false

2. Create main.go

Added /v1/greeter API.

// Copyright (c) 2021 rookie-ninja
//
// Use of this source code is governed by an Apache-style
// license that can be found in the LICENSE file.

package main

import (
	"context"
	"fmt"
	"github.com/rookie-ninja/rk-boot"
	"github.com/rookie-ninja/rk-boot/mux"
	"github.com/rookie-ninja/rk-mux/interceptor"
	"net/http"
)

func main() {
	// Create a new boot instance.
	boot := rkboot.NewBoot()

	// Register handler
	entry := rkbootmux.GetMuxEntry("greeter")
	entry.Router.NewRoute().Methods(http.MethodGet).Path("/v1/greeter").HandlerFunc(Greeter)

	// Bootstrap
	boot.Bootstrap(context.TODO())

	boot.WaitForShutdownSig(context.TODO())
}

func Greeter(writer http.ResponseWriter, request *http.Request) {
	rkmuxinter.WriteJson(writer, http.StatusOK, &GreeterResponse{
		Message: fmt.Sprintf("Hello %s!", request.URL.Query().Get("name")),
	})
}

// Response.
type GreeterResponse struct {
	Message string
}

3. Folder structure

$ tree
.
├── boot.yaml
├── go.mod
├── go.sum
└── main.go

0 directories, 4 files

4. Start main.go

$ go run main.go

2022-02-09T15:35:02.181+0800    INFO    boot/mux_entry.go:643   Bootstrap muxEntry      {"eventId": "a35a0331-4311-4057-a399-526c76f79ca9", "entryName": "greeter", "entryType": "Mux"}
------------------------------------------------------------------------
endTime=2022-02-09T15:35:02.181722+08:00
startTime=2022-02-09T15:35:02.181528+08:00
elapsedNano=193785
timezone=CST
ids={"eventId":"a35a0331-4311-4057-a399-526c76f79ca9"}
app={"appName":"rk","appVersion":"","entryName":"greeter","entryType":"Mux"}
env={"arch":"amd64","az":"*","domain":"*","hostname":"lark.local","localIP":"192.168.1.102","os":"darwin","realm":"*","region":"*"}
payloads={"commonServiceEnabled":true,"commonServicePathPrefix":"/rk/v1/","muxPort":8080,"promEnabled":true,"promPath":"/metrics","promPort":8080}
counters={}
pairs={}
timing={}
remoteAddr=localhost
operation=Bootstrap
resCode=OK
eventStatus=Ended
EOE

5. Verify

• Send a request to the /rk/v1/healthy API in CommonService.

$ curl -X GET localhost:8080/rk/v1/healthy
{"healthy":true}

• Send a request to the /v1/greeter API.

$ curl -X GET "localhost:8080/v1/greeter?name=rk-dev"
{"Message":"Hello rk-dev!"}

Access the Prometheus client: http://localhost:8080/metrics

Visual monitoring

We've started prometheus monitoring in the local process, and all that's left is how to view the monitoring in a [nice] web page.

There are many tools on the market, but we choose the [simple], [popular], and [free] methods, that is, Prometheus + Grafana.

Architecture diagram

Let's take a look at what the whole process looks like.

In fact, the principle is very simple, it is to [hijack] API requests, and record [time], [error code] and other information. After that, let the Prometheus service actively pull data from the [created service]. Finally, let the Grafana service pull data from Prometheus and display the data table.

quick start

1. Create prometheus.yml

Let's first create the prometheus.yml configuration file so that the prometheus service can pull data from localhost:8080/metrics.

In the configuration below, we do not specify /metrics, because prometheus uses /metrics to pull data by default.

Notice! We set targets to host.docker.internal:8080 instead of localhost:8080, this is because prometheus is in the container and our service is in the local.

This is a convenient way to access the local machine's port from within a container. explain

global:
  scrape_interval: 1s # Make scrape interval to 1s for testing.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  - job_name: 'rk-demo'
    scrape_interval: 1s
    static_configs:
      - targets: ['host.docker.internal:8080']

2. Start Prometheus

We use docker to start.

Prometheus uses port 9090 by default.

$ docker run -p 9090:9090 -v /<your path>/rk-demo/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

3. Verify Prometheus

Please follow the above [Verification], start main.go, and send a /rk/v1/healthy request.

Then, let's take a look at the data in the prometheus service.

Visit: localhost:9090, and search for rk_greeter_resCode, we can see that the data is already in prometheus.

Visit: localhost:9090/targets, we can see that prometheus has pulled data every second.

4. Start Grafana

Grafana uses port 3000 by default.

$ docker run -p 3000:3000 --name grafana grafana/grafana

Access: localhost:3000

At this time, grafana will let you log in. The default username and password are as follows.

Username: admin Password: admin

5. Add Prometheus data source in Grafana

Grafana is just a web UI tool, in order to see the data report, we tell Grafana where to look for Prometheus.

Select Prometheus as the data source.

Fill in the Prometheus address, the same as above, because Grafana runs in Docker, so we don't use localhost:9090, but host.docker.internal:9090.

6. Import Dashboard

We can edit the Grafana Dashboard by ourselves, but this is not an easy task. For services started with rk-boot, we provide the default [free] Grafana Dashboard template.

Note that the Dashboard imported here only matches [services created according to the above logic].

Why? Because rk-boot will use rk_<Entry name>_xxx as the metrics name of prometheus by default.

Move to Dashboard import page

Import Dashboard No. 15111, defined at: https://grafana.com/grafana/dashboards/15111

Specify the Prometheus data source, which is the Prometheus we configured above.

start monitoring

Notice! If the number of requests is too small, it will be displayed as 0, please send several more requests.

concept

We can already get monitoring data from Grafana, now let's look at the middleware in rk-boot, what type of monitoring data is added.

The monitoring interceptor will record the following monitoring by default.

Monitoring item	type of data	Details
elapsedNano	Summary	RPC time consuming
resCode	Counter	Counters based on RPC return codes
errors	Counter	RPC error based counters

The above three monitoring items have the following labels.

Label	Details
entryName	Mux entry name
entryType	Mux entry type
realm	Environment variables: REALM, eg: rk
region	Environment variables: REGION, eg: beijing
the	Environment variables: AZ, eg: beijing-1
domain	Environment variables: DOMAIN, eg: prod
instance	local hostname
appVersion	Get from AppInfoEntry
appName	Get from AppInfoEntry
restMethod	Http method. eg: GET
restPath	Http path. eg: /rk/v1/healthy
type	Service type. eg: Mux
resCode	Return code, eg: OK

{{o.name}}

{{m.name}}