Observability: Manual instrumentation of Go applications using OpenTelemetry

Enterprise 2023-09-30 09:51:58 views: null

Author: Luca Wintergerst

DevOps and SRE teams are changing the software development process. DevOps engineers focus on efficient software application and service delivery, while SRE teams are key to ensuring reliability, scalability, and performance. These teams must rely on full-stack observability solutions that enable them to manage and monitor systems and ensure issues are resolved before they impact the business.

Observability across the entire modern distributed application stack requires collecting, processing, and correlating data, often in the form of dashboards. Ingesting all system data requires installing agents across stacks, frameworks, and providers, a process that can be challenging and time-consuming for teams that must deal with version changes, compatibility issues, and proprietary code that does not scale with system changes.

Thanks to OpenTelemetry (OTel) , DevOps and SRE teams now have a standard way to collect and send data that doesn't rely on proprietary code and has large community support, reducing vendor lock-in.

In this blog post, we will show you how to manually instrument a Go application using OpenTelemetry. This method is slightly more complex than using automatic detection

In previous blogs , we also reviewed how to use the OpenTelemetry demo and connect it to Elastic®, as well as some of the features of Elastic and OpenTelemetry. In this blog, we will use another demo application that helps to highlight manual detection in a simple way.

Finally, we'll discuss how Elastic supports mixed-mode applications running with Elastic and OpenTelemetry agents. The advantage of this is that no  hotel-collector is required ! This setup enables you to slowly and easily migrate applications to OTel using Elastic based on the schedule that works best for your business.

Applications, prerequisites and configuration

The app we're using in this blog is called Elastiflix , a movie streaming app. It consists of multiple microservices written in .NET, NodeJS, Go, and Python.

Before we instrument the sample application, we first need to understand how Elastic receives telemetry data.

Elastic configuration options for OpenTelemetry

All APM capabilities of Elastic Observability are available through OTel data. Some of these include:

  • Service maps
  • Service details (latency, throughput, failed transactions)
  • Dependencies between services, distributed tracing
  • transactions (trace)
  • Machine Learning (ML) Relevance
  • Log correlation

In addition to Elastic's unified view of APM and telemetry data, you can use Elastic's powerful machine learning capabilities to reduce analytics and generate alerts to help reduce MTTR.

prerequisites

View sample source code

The complete source code, including the Dockerfile used in this blog, can be found on GitHub. This repository also contains the same application, but without instrumentation. This allows you to compare each file and see the differences.

Before we begin, let's take a look at the uninstrumented code.

This is our simple go application that can receive GET requests. Note that the code shown here is a slightly abbreviated version.

package main

import (
	"log"
	"net/http"
	"os"
	"time"

	"github.com/go-redis/redis/v8"

	"github.com/sirupsen/logrus"

	"github.com/gin-gonic/gin"
	"strconv"
	"math/rand"
)

var logger = &logrus.Logger{
	Out:   os.Stderr,
	Hooks: make(logrus.LevelHooks),
	Level: logrus.InfoLevel,
	Formatter: &logrus.JSONFormatter{
		FieldMap: logrus.FieldMap{
			logrus.FieldKeyTime:  "@timestamp",
			logrus.FieldKeyLevel: "log.level",
			logrus.FieldKeyMsg:   "message",
			logrus.FieldKeyFunc:  "function.name", // non-ECS
		},
		TimestampFormat: time.RFC3339Nano,
	},
}

func main() {
	delayTime, _ := strconv.Atoi(os.Getenv("TOGGLE_SERVICE_DELAY"))

	redisHost := os.Getenv("REDIS_HOST")
	if redisHost == "" {
		redisHost = "localhost"
	}

	redisPort := os.Getenv("REDIS_PORT")
	if redisPort == "" {
		redisPort = "6379"
	}

	applicationPort := os.Getenv("APPLICATION_PORT")
	if applicationPort == "" {
		applicationPort = "5000"
	}

	// Initialize Redis client
	rdb := redis.NewClient(&redis.Options{
		Addr:     redisHost + ":" + redisPort,
		Password: "",
		DB:       0,
	})

	// Initialize router
	r := gin.New()
	r.Use(logrusMiddleware)

	r.GET("/favorites", func(c *gin.Context) {
		// artificial sleep for delayTime
		time.Sleep(time.Duration(delayTime) * time.Millisecond)

		userID := c.Query("user_id")

		contextLogger(c).Infof("Getting favorites for user %q", userID)

		favorites, err := rdb.SMembers(c.Request.Context(), userID).Result()
		if err != nil {
			contextLogger(c).Error("Failed to get favorites for user %q", userID)
			c.String(http.StatusInternalServerError, "Failed to get favorites")
			return
		}

		contextLogger(c).Infof("User %q has favorites %q", userID, favorites)

		c.JSON(http.StatusOK, gin.H{
			"favorites": favorites,
		})
	})

	// Start server
	logger.Infof("App startup")
	log.Fatal(http.ListenAndServe(":"+applicationPort, r))
	logger.Infof("App stopped")
}

Step-by-step guide

Step 0. Sign in to your Elastic Cloud account

This blog assumes you have an Elastic Cloud account - if not, follow the instructions to get started with Elastic Cloud .

Step 1. Install and initialize OpenTelemetry

As a first step, we need to add some additional packages to the application.

import (
      "github.com/go-redis/redis/extra/redisotel/v8"
      "go.opentelemetry.io/otel"
      "go.opentelemetry.io/otel/attribute"
      "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

	"go.opentelemetry.io/otel/propagation"

	"google.golang.org/grpc/credentials"
	"crypto/tls"

      sdktrace "go.opentelemetry.io/otel/sdk/trace"

	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

	"go.opentelemetry.io/otel/trace"
	"go.opentelemetry.io/otel/codes"
)

This code imports the necessary OpenTelemetry packages, including those for tracing, exporting, and instrumenting specific libraries such as Redis.

Next we read the OTEL_EXPORTER_OTLP_ENDPOINT variable and initialize the exporter.

var (
    collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
)
var tracer trace.Tracer


func initTracer() func(context.Context) error {
	tracer = otel.Tracer("go-favorite-otel-manual")

	// remove https:// from the collector URL if it exists
	collectorURL = strings.Replace(collectorURL, "https://", "", 1)
	secretToken := os.Getenv("ELASTIC_APM_SECRET_TOKEN")
	if secretToken == "" {
		log.Fatal("ELASTIC_APM_SECRET_TOKEN is required")
	}

	secureOption := otlptracegrpc.WithInsecure()
    exporter, err := otlptrace.New(
        context.Background(),
        otlptracegrpc.NewClient(
            secureOption,
            otlptracegrpc.WithEndpoint(collectorURL),
			otlptracegrpc.WithHeaders(map[string]string{
				"Authorization": "Bearer " + secretToken,
			}),
			otlptracegrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})),
        ),
    )

    if err != nil {
        log.Fatal(err)
    }

    otel.SetTracerProvider(
        sdktrace.NewTracerProvider(
            sdktrace.WithSampler(sdktrace.AlwaysSample()),
            sdktrace.WithBatcher(exporter),
        ),
    )
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.Baggage{},
			propagation.TraceContext{},
		),
	)
    return exporter.Shutdown
}

To detect Redis connections, we will add a tracing hook to it, and to detect Gin, we will add OTel middleware. This will automatically capture all interactions with our application as Gin will be fully instrumented. Additionally, all outgoing connections to Redis will also be inspected.

      // Initialize Redis client
	rdb := redis.NewClient(&redis.Options{
		Addr:     redisHost + ":" + redisPort,
		Password: "",
		DB:       0,
	})
	rdb.AddHook(redisotel.NewTracingHook())
	// Initialize router
	r := gin.New()
	r.Use(logrusMiddleware)
	r.Use(otelgin.Middleware("go-favorite-otel-manual"))

Add custom span

Now that we have everything added and initialized, we can add custom spans.

If we want to provide additional instrumentation for a part of the application, we can simply start a custom span and defer ending the span.


// start otel span
ctx := c.Request.Context()
ctx, span := tracer.Start(ctx, "add_favorite_movies")
defer span.End()

For comparison, here is the instrumentation code for our sample application. You can find the complete source code on GitHub.

package main

import (
	"log"
	"net/http"
	"os"
	"time"
	"context"

	"github.com/go-redis/redis/v8"
	"github.com/go-redis/redis/extra/redisotel/v8"


	"github.com/sirupsen/logrus"

	"github.com/gin-gonic/gin"

  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/attribute"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

	"go.opentelemetry.io/otel/propagation"

	"google.golang.org/grpc/credentials"
	"crypto/tls"

  sdktrace "go.opentelemetry.io/otel/sdk/trace"

	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"

	"go.opentelemetry.io/otel/trace"
	
	"strings"
	"strconv"
	"math/rand"
	"go.opentelemetry.io/otel/codes"

)

var tracer trace.Tracer

func initTracer() func(context.Context) error {
	tracer = otel.Tracer("go-favorite-otel-manual")

	collectorURL = strings.Replace(collectorURL, "https://", "", 1)

	secureOption := otlptracegrpc.WithInsecure()

	// split otlpHeaders by comma and convert to map
	headers := make(map[string]string)
	for _, header := range strings.Split(otlpHeaders, ",") {
		headerParts := strings.Split(header, "=")

		if len(headerParts) == 2 {
			headers[headerParts[0]] = headerParts[1]
		}
	}

    exporter, err := otlptrace.New(
        context.Background(),
        otlptracegrpc.NewClient(
            secureOption,
            otlptracegrpc.WithEndpoint(collectorURL),
			otlptracegrpc.WithHeaders(headers),
			otlptracegrpc.WithTLSCredentials(credentials.NewTLS(&tls.Config{})),
        ),
    )

    if err != nil {
        log.Fatal(err)
    }

    otel.SetTracerProvider(
        sdktrace.NewTracerProvider(
            sdktrace.WithSampler(sdktrace.AlwaysSample()),
            sdktrace.WithBatcher(exporter),
            //sdktrace.WithResource(resources),
        ),
    )
	otel.SetTextMapPropagator(
		propagation.NewCompositeTextMapPropagator(
			propagation.Baggage{},
			propagation.TraceContext{},
		),
	)
    return exporter.Shutdown
}

var (
  collectorURL = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
	otlpHeaders = os.Getenv("OTEL_EXPORTER_OTLP_HEADERS")
)


var logger = &logrus.Logger{
	Out:   os.Stderr,
	Hooks: make(logrus.LevelHooks),
	Level: logrus.InfoLevel,
	Formatter: &logrus.JSONFormatter{
		FieldMap: logrus.FieldMap{
			logrus.FieldKeyTime:  "@timestamp",
			logrus.FieldKeyLevel: "log.level",
			logrus.FieldKeyMsg:   "message",
			logrus.FieldKeyFunc:  "function.name", // non-ECS
		},
		TimestampFormat: time.RFC3339Nano,
	},
}

func main() {
	cleanup := initTracer()
  defer cleanup(context.Background())

	redisHost := os.Getenv("REDIS_HOST")
	if redisHost == "" {
		redisHost = "localhost"
	}

	redisPort := os.Getenv("REDIS_PORT")
	if redisPort == "" {
		redisPort = "6379"
	}

	applicationPort := os.Getenv("APPLICATION_PORT")
	if applicationPort == "" {
		applicationPort = "5000"
	}

	// Initialize Redis client
	rdb := redis.NewClient(&redis.Options{
		Addr:     redisHost + ":" + redisPort,
		Password: "",
		DB:       0,
	})
	rdb.AddHook(redisotel.NewTracingHook())


	// Initialize router
	r := gin.New()
	r.Use(logrusMiddleware)
	r.Use(otelgin.Middleware("go-favorite-otel-manual"))

	
	// Define routes
	r.GET("/", func(c *gin.Context) {
		contextLogger(c).Infof("Main request successful")
		c.String(http.StatusOK, "Hello World!")
	})

	r.GET("/favorites", func(c *gin.Context) {
		// artificial sleep for delayTime
		time.Sleep(time.Duration(delayTime) * time.Millisecond)
		
		userID := c.Query("user_id")

		contextLogger(c).Infof("Getting favorites for user %q", userID)

		favorites, err := rdb.SMembers(c.Request.Context(), userID).Result()
		if err != nil {
			contextLogger(c).Error("Failed to get favorites for user %q", userID)
			c.String(http.StatusInternalServerError, "Failed to get favorites")
			return
		}

		contextLogger(c).Infof("User %q has favorites %q", userID, favorites)

		c.JSON(http.StatusOK, gin.H{
			"favorites": favorites,
		})
	})

	// Start server
	logger.Infof("App startup")
	log.Fatal(http.ListenAndServe(":"+applicationPort, r))
	logger.Infof("App stopped")
}

Step 2. Run the Docker image using environment variables

As specified in the OTEL documentation, we will use environment variables and pass in the configuration values ​​found in the APM agent configuration section.

Since Elastic natively accepts OTLP, we only need to provide the endpoint and authentication that the OTEL Exporter needs to send data, as well as some other environment variables.

Where to get these variables in Elastic Cloud and Kibana®

You can copy the endpoint and token from Kibana under the path /app/home#/tutorial/apm.

You need to copy OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS.

Build image

docker build -t  go-otel-manual-image .

Run image

docker run \
       -e OTEL_EXPORTER_OTLP_ENDPOINT="<REPLACE WITH OTEL_EXPORTER_OTLP_ENDPOINT>" \
       -e OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <REPLACE WITH TOKEN>" \
       -e OTEL_RESOURCE_ATTRIBUTES="service.version=1.0,deployment.environment=production,service.name=go-favorite-otel-manual" \
       -p 5000:5000 \
       go-otel-manual-image

You can now make some requests to generate tracking data. Note that these requests are expected to return errors because this service relies on a Redis connection that you are not currently running. As mentioned before, you can find a more complete example using Docker compose here .

curl localhost:500/favorites
# or alternatively issue a request every second

while true; do curl "localhost:5000/favorites"; sleep 1; done;

How do traces appear in Elastic?

Now that the service has been instrumented, you should see the following output in Elastic APM when viewing the transactions section of the Node.js service:

in conclusion

In this blog we discuss the following:

  • How to manually detect Go using OpenTelemetry
  • How to properly initialize OpenTelemetry and add custom scopes
  • How to easily set up OTLP ENDPOINT and OTLP HEADERS using Elastic without the need for a collector

Hopefully it provides an easy-to-understand walkthrough of instrumenting Go using OpenTelemetry, and how easy it is to send traces to Elastic.

Additional resources about OpenTelemetry with Elastic:

Developer resources:


General configuration and use case resources:

原文：Manual instrumentation of Go applications with OpenTelemetry | Elastic Blog

Guess you like

Origin blog.csdn.net/UbuntuTouch/article/details/132883636
Recommended
Ranking
Goldman: Microsoft is winning the war cloud services
Change admob ids on existing apps
【C++】Introduction to Object-Oriented Programming③ (Process-Oriented Programming Structured Programming Method | Structured Programming Method Concept / Features / Advantages and Disadvantages | Object-Oriented Programming Introduction )
2021-02-26 Luogu P1966 digging landmines-dfs violent search (backtracking method)
jqgrid seleccione todo cancelar una sola fila haga clic en cancelar evento
InstructTTS: Modeling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
hdu1312 “Red and Black”——DFS与递归的应用
【Linux】孤儿进程、僵死进程和信号
OCRアーティファクト、PDF、数式を変換可能
Linux kernel Tcp performance tuning
Daily
More
2024-06-04(0)
2024-06-03(1)
2024-06-02(0)
2024-06-01(1)
2024-05-31(1)
2024-05-30(0)
2024-05-29(1)
2024-05-28(1)
2024-05-27(1)
2024-05-26(0)

Code World

© 2018 codetd.com