Example of jaeger-operator deployment in production environment

background

The installation package of istio comes with jaeger deployment method, but the deployment method it provides is all-in-one, that is, the data storage is in memory, which is not suitable for the production environment. Therefore, we need to follow the official Deploy in the production environment deployment method. This is also explained in the official documentation of istio.

Insert image description here
In kubernetes, the official recommendation is to use the operator method for deployment. The deployment method is not difficult. Just refer to the official documentation. However, there are some things that need to be paid attention to when connecting to istio. I will record my installation process below.

This article only records how to deploy jaeger in a production environment. Regarding the in-depth application of jaeger, I will not introduce it here. Students who need it can refer to the official documentation by themselves.

Reference documentation: jaeger-operator

deploy

Deploy jaeger-operator

According to the official documentation, install cert-manager first
! [Insert image description here](https://img-blog.csdnimg.cn/e96a63cd30924082a35cf0e6571af723.png

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.6.3/cert-manager.yaml

Deploy jaeger-operator

According to the official document, continue the operation.

Because 1.46.0 has not yet been released on github, the execution of the command according to the official document will fail, so we can settle for the next best thing and install 1.45.0


kubectl create ns observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.45.0/jaeger-operator.yaml -n observability 
kubectl get deployment jaeger-operator -n observability

NAME              DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jaeger-operator   1         1         1            1           48s

Note that pod startup failure may occur here. When you do this kubectl describe pod -n observability jaeger-operator-58d97648c5-****, you will find that gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0 failed to download. Those who often play K8S should know that the Google library is blocked. , you need to find a way to get this image by yourself. You can pull it from the domestic image address and retag it, or download it in a ladder environment and export and import the image... In short, there are many ways.

Deploy jaeger

Because everyone faces different environments in this step, you need to customize the configuration according to your actual environment. It is recommended to read the document several times.

Here are my needs: The storage I use is an external ES. At the same time, I refer to the official configuration and use the Elasticsearch rollover configuration method. Then the external ES has an account and password, so I can use Secrets Support to configure the account and password. The details of secrets are For configuration methods, please refer to the fourth description of External Elasticsearchkubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic
Insert image description here
. My configuration is posted below.

#创建elasticsearch的用户密码secrets
$ kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic
$ kubectl get secrets -n observability 
NAME                           TYPE                                  DATA   AGE
default-token-fc756            kubernetes.io/service-account-token   3      23h
jaeger-operator-service-cert   kubernetes.io/tls                     3      23h
jaeger-operator-token-dmrzc    kubernetes.io/service-account-token   3      20h
jaeger-secret                  Opaque                                2      22h
jaeger-token-n5xfx             kubernetes.io/service-account-token   3      20h

##jaeger部署文件
$ cat jaeger.yaml 
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: production
  collector:
    maxReplicas: 2
    resources:
      limits:
        cpu: 100m
        memory: 128Mi
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://192.168.8.105:9200
        use-aliases: true
    esRollover:
      conditions: "{
    
    \"max_age\": \"2d\"}"
      readTTL: 168h
      schedule: "55 23 * * *"
    secretName: jaeger-secret
  query:
    serviceType: NodePort
$ kubectl apply -f jaeger.yaml
$ kubectl get svc,pod -n observability 
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                    AGE
service/jaeger-collector                  ClusterIP   10.233.19.93    <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   20h
service/jaeger-collector-headless         ClusterIP   None            <none>        9411/TCP,14250/TCP,14267/TCP,14268/TCP,4317/TCP,4318/TCP   20h
service/jaeger-operator-metrics           ClusterIP   10.233.29.204   <none>        8443/TCP                                                   20h
service/jaeger-operator-webhook-service   ClusterIP   10.233.28.228   <none>        443/TCP                                                    20h
service/jaeger-query                      NodePort    10.233.23.105   <none>        16686:32003/TCP,16685:32004/TCP                            20h

NAME                                           READY   STATUS      RESTARTS   AGE
pod/jaeger-collector-c498bfb45-khtrq           1/1     Running     0          20h
pod/jaeger-es-index-cleaner-28102555-t4v77     0/1     Completed   0          14h
pod/jaeger-es-lookback-28102555-d98x8          0/1     Completed   0          14h
pod/jaeger-es-rollover-28102555-2rxlw          0/1     Completed   0          14h
pod/jaeger-es-rollover-create-mapping-k4x5r    0/1     Completed   0          20h
pod/jaeger-operator-58d97648c5-gr2kx           2/2     Running     0          20h
pod/jaeger-query-79754974c7-7gnk9              2/2     Running     0          20h
pod/jaeger-spark-dependencies-28102555-dbnxt   0/1     Completed   0          14h

docking istio

So far, the basic deployment has been completed. You can already use the WEB-UI provided by jaeger-query, but there is no istio-related tracing information at this time. We have mentioned it before when we read the istio official document. The address of jaeger-collector needs to be defined in the istio configuration. The official method is to define the parameters when installing (updating) istio. In fact, there is another method:

$ kubectl get cm -n istio-system 
NAME                                  DATA   AGE
grafana                               4      63d
istio                                 2      63d
istio-ca-root-cert                    1      63d
istio-gateway-deployment-leader       0      63d
istio-gateway-status-leader           0      63d
istio-grafana-dashboards              2      63d
istio-leader                          0      63d
istio-namespace-controller-election   0      63d
istio-services-grafana-dashboards     4      63d
istio-sidecar-injector                2      63d
kiali                                 1      63d
kube-root-ca.crt                      1      63d
prometheus                            5      63d

istiod has a cm, and its name is istio. We only need to add or modify it in istio
Insert image description here
. It does not mean that it can be used normally. At this time, you need to restart istiod, and then you will find that the service you deployed previously The tracing information still cannot be seen through jaeger. At this time, you need to restart the service injected by istio, and then you can see the tracing information.
Insert image description here

dockingkiali

Students who have used kiali know that traces can be viewed in several menus of kiali, and the data source is jaeger. If we use the jaeger that comes with istio, we can view it directly without any configuration. But now we deploy jaejer ourselves
Insert image description here . , which is slightly different from the installation method that comes with istio, so we need to configure it. The specific method is also very simple, which is to modify the configmap of kiali

$ kubectl get cm -n istio-system kiali 
NAME    DATA   AGE
kiali   1      63d

Then find external_servicesand add the following content below

  tracing:
    url: http://192.168.8.104:32003
    in_cluster_url: http://jaeger-query.observability:16685/jaeger

Insert image description here
The url is equivalent to the address of your external access to jaeger's webUI. Its function is to allow you to directly access jaeger in kiali. in_cluster_url is to display traces in kiali.