Monitor Kubernetes performance with Heapster and Splunk

Kubernetes has become the de facto king of container orchestration, and even Docker has bowed to the Queen of K8s. Data collection and monitoring of Kubernetes clusters has become an important topic in IT operation and maintenance. Let's take a look today at how to use Splunk's latest Metrics Store for performance monitoring of Kubernetes clusters.

Deployment Architecture

The following figure is the deployment architecture of the solution, which mainly includes:

Use Heapster to collect K8s performance data, including CPU, Memory, Network, File System, etc.
Use Heapster's Statsd Sink to send data to Splunk's Metrics Store
Monitor performance data with Splunk's search commands and dashboards

Preliminary preparation

There are two main things to prepare in advance:

Compile the latest Heapster image and upload it to a public Docker image repository, such as docker hub
Configure Metrics Store and corresponding network input (Network Input UDP/TCP) in Splunk

The main choice here is whether to use UDP or TCP for the transmission protocol of Statsd. Here I recommend using TCP. The latest Heapster code supports different backends, including log, influxdb, stackdriver, gcp monitoring, gcp logging, statsd, hawkular-metrics, wavefront, openTSDB, kafka, riemann, elasticsearch and more. Because Splunk's Metrics Store supports the statsd protocol, it can be easily integrated with Heapster.

First, we need to use the latest heapster code to compile a container image, because the official image version of heapsterd on docker hub is older and does not support statsd. So you need to compile it yourself.

mkdir myheapster
mkdir myheapster/src
export GOPATH=myheapster
cd myheapster/src
git clone https://github.com/kubernetes/heapster.git
cd heapster
make container

Run the above command to compile the latest heapster image.

Note that heapster uses udp protocol by default. If you want to use tcp, you need to modify the code

https://github.com/kubernetes/heapster/blob/master/metrics/sinks/statsd/statsd_client.go

func (client *statsdClientImpl) open() error {
	var err error
	client.conn, err = net.Dial("udp", client.host)
	if err != nil {
		glog.Errorf("Failed to open statsd client connection : %v", err)
	} else {
		glog.V(2).Infof("statsd client connection opened : %+v", client.conn)
	}
	return err
}

Change udp to tcp.

I put two images on the docker hub, corresponding to the tcp version of the udp version, you can use it directly

naughtytao/heapster-amd64:v1.5.0-beta.3 udp
naughtytao/heapster-amd64:v1.5.0-beta.4 tcp

Then you need to configure the Metrics Store in Splunk, refer to this document

Install and configure Heapster

It is easier to deploy heapster on K8s, create the corresponding yaml configuration file, and then use the kubectl command line to create it.

The following are the configuration files for Deployment and Service:

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
        version: v6
    spec:
      containers:
      - name: heapster
        image: naughtytao/heapster-amd64:v1.5.0-beta.3
        imagePullPolicy: Always
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=statsd:udp://ip:port?numMetricsPerMsg=1

service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

Note the --sink configuration of deployment here, ip is the IP or hostname of Splunk, and port corresponds to the port number of Splunk's data input. When using the udp protocol, the value of numMetricsPerMsg that needs to be configured is relatively small, and when the value is relatively large, an error of message too long will occur . A larger value can be configured when using tcp.

Run kubectl apply -f *.yaml to deploy heapster

If it is running normally, the log of the corresponding heapster pod is as follows

I0117 18:10:56.054746       1 heapster.go:78] /heapster --source=kubernetes:https://kubernetes.default --sink=statsd:udp://ec2-34-203-25-154.compute-1.amazonaws.com:8124?numMetricsPerMsg=10
I0117 18:10:56.054776       1 heapster.go:79] Heapster version v1.5.0-beta.4
I0117 18:10:56.054963       1 configs.go:61] Using Kubernetes client with master "https://kubernetes.default" and version v1
I0117 18:10:56.054978       1 configs.go:62] Using kubelet port 10255
I0117 18:10:56.076200       1 driver.go:104] statsd metrics sink using configuration : {host:ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15fc8c0}
I0117 18:10:56.076248       1 driver.go:104] statsd metrics sink using configuration : {host:ec2-34-203-25-154.compute-1.amazonaws.com:8124 prefix: numMetricsPerMsg:10 protocolType:etsystatsd renameLabels:map[] allowedLabels:map[] customizeLabel:0x15fc8c0}
I0117 18:10:56.076272       1 heapster.go:202] Starting with StatsD Sink
I0117 18:10:56.076281       1 heapster.go:202] Starting with Metric Sink
I0117 18:10:56.090229       1 heapster.go:112] Starting heapster on port 8082

Monitoring in Splunk

Well, if everything is normal, heapster will send metrics to Splunk's metrics store using the statsd protocol and format.

Then you can use the mstats and mcatalog commands of SPL to analyze and monitor the metrics data.

The following search statement lists all metrics

| mcatalog values(metric_name)

The following search statement lists the CPU usage of the entire cluster, and we can use Area or Line Chart to visualize the search results.

| mstats avg(_value) WHERE metric_name=cluster.cpu/usage_rate span=30m

Corresponding memory usage of kube-system namespace

| mstats avg(_value) WHERE metric_name=namespace.kube-system.memory/usage span=30m

You can put the analysis results you are interested in on the Dashboard and use the Realtime settings to monitor.

Well, more analysis options can refer to the Splunk documentation.