kubernetes部署skywalking集群和JAVA服务的接入

1 概述：

1.1 环境

版本信息如下：
a、操作系统：centos 7.6
a、skywalking版本：v8.0.1
c、kubernetes版本：v1.18.5
d、es版本：6.4.3
e、helm版本： helm3

1.2 skywalking概述

1.2.1 skywalking是什么

SkyWalking是一个开源的APM系统，为云原生分布式系统提供监控、链路追踪、诊断能力，支持集成多种编程语言的应用（java、php、go、lua等），也能和服务网格进行集成。除了支持代码侵入方式的集成，一个主要亮点也支持零代码入侵的集成（零代码侵入是和具体的编程语言相关的），是利用java agent的特性在jvm级别修改了运行时的程序，因此程序员在代码编辑期间不需要修改业务代码也能达到埋点的效果。后端存储支持es、mysql、tidb等多种数据库。
架构图如下：

1.2.1 skywalking的java代理的使用

1）方式1：命令行方式

java \
-javaagent:/root/skywalking/agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=app1 \
-Dskywalking.collector.backend_service=localhost:11800 \
-jar myapp.jar

2）方式2：环境变量方式

export SW_AGENT_COLLECTOR_BACKEND_SERVICES=10.0.0.1:11800,10.0.0.2:11800
export SW_AGENT_NAME=demo1
export JAVA_OPTS=-javaagent:/root/skywalking/agent/skywalking-agent.jar

java \
$JAVA_OPTS \
-jar myapp.jar

2 部署前置条件：

具备一个k8s集群：

3 部署：

3.1 部署es集群

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  namespace: kube-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: elasticsearch
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch
    spec:
      containers:
      - env:
        - name: cluster.name
          value: k8s-logs
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: discovery.zen.ping.unicast.hosts
          value: elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch
        - name: discovery.zen.minimum_master_nodes
          value: "2"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/elasticsearch-oss:6.4.3
        imagePullPolicy: Always
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 100m
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      initContainers:
      - command:
        - sh
        - -c
        - chown -R 1000:1000 /usr/share/elasticsearch/data
        image: busybox
        imagePullPolicy: Always
        name: fix-permissions
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /usr/share/elasticsearch/data
          name: data
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: busybox
        imagePullPolicy: Always
        name: increase-vm-max-map
        resources: {}
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - command:
        - sh
        - -c
        - ulimit -n 65536
        image: busybox
        imagePullPolicy: Always
        name: increase-fd-ulimit
        resources: {}
        securityContext:
          privileged: true
      volumes:
      - emptyDir: {}
        name: data

---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: kube-system
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node

---
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - port: 9200
      name: external

es集群的方式地址是：192.168.39.33:18448

3.2 部署skywalking集群

从github中下载skywalking的chart包仓库

cd /tmp
rpm -q git > /dev/null 2>&1 || yum install git -y
git clone https://github.com/apache/skywalking-kubernetes
cd /tmp/skywalking-kubernetes/chart

由于已存在es集群，因此不需要再通过helm去部署es。可把chart包依赖的es chart注释掉。

vim skywalking/Chart.yaml

执行helm命令部署skywalking集群。在第一步，我已经在kube-system名字空间下部署了es集群，因此skywalking连接的es集群是：elasticsearch-logging.kube-system:9200。

export SKYWALKING_RELEASE_NAME=skywalking
export SKYWALKING_RELEASE_NAMESPACE=skywalking-ns
kubectl create ns $SKYWALKING_RELEASE_NAMESPACE

helm install "$SKYWALKING_RELEASE_NAME" ./skywalking \
  -n "$SKYWALKING_RELEASE_NAMESPACE" \
  --set oap.image.tag=8.0.1-es6 \
  --set oap.storageType=elasticsearch \
  --set oap.service.type=NodePort \
  --set oap.javaOpts="-Xmx8g -Xms8g" \
  --set ui.image.tag=8.0.1 \
  --set ui.service.type=NodePort \
  --set elasticsearch.enabled=false \
  --set elasticsearch.config.host=elasticsearch-logging.kube-system \
  --set elasticsearch.config.port.http=9200 \
  --set elasticsearch.config.user="" \
  --set elasticsearch.config.password=""  

查看svc和pod，可见部署skywalking成功：

通过k8s svc NodePort得知，skywalking agent连接的后端服务oap地址是：192.168.39.33:22335。

3.3 制作skywalking agent的init容器

cd /tmp/
wget https://archive.apache.org/dist/skywalking/8.0.1/apache-skywalking-apm-8.0.1.tar.gz
tar xf /apache-skywalking-apm-8.0.1.tar.gz

扫描二维码关注公众号，回复： 12950606 查看本文章

编写Dockerfile：如下：

FROM busybox:latest
ENV LANG=C.UTF-8
RUN set -eux && mkdir -p /opt/skywalking/agent/
ADD apache-skywalking-apm-bin/agent/ /opt/skywalking/agent/
WORKDIR /

执行docker build命令制作镜像，并推送至仓库。
docker build -t “registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1” .
docker push registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1

4 部署springboot微服务

1）微服务来自网上，并做了一些修改。微服务几乎没有业务逻辑，只有http调用和睡眠指令。
2）我的业务服务部署在另外一个k8s集群中，因此skywalking agent访问的是位于另一个集群中的skywalking oap服务的NodePort。
3）每个yaml文件都可以直接使用，需要根据实际情况修改环境变量SW_AGENT_COLLECTOR_BACKEND_SERVICES。在我的例子中SW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.39.33:22335。

4.1 UI服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-ui
  name: acme-financial-ui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-ui
  template:
    metadata:
      labels:
        app: acme-financial-ui
    spec:
      initContainers:
      - image: registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-ui"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.168.30.68:22335"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-ui:v0.1
        imagePullPolicy: Always
        name: ui
        ports:
        - containerPort: 8081
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent      
      volumes: 
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-ui
  name: acme-financial-ui
spec:
  ports:
  - name: http
    port: 8081
    protocol: TCP
    targetPort: 8081
  selector:
    app: acme-financial-ui
  sessionAffinity: None
  type: NodePort

4.2 office服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-office
  name: acme-financial-office
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-office
  template:
    metadata:
      labels:
        app: acme-financial-office
    spec:
      initContainers:
      - image: registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-office"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.168.30.68:22335"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-office:v0.1
        imagePullPolicy: Always
        name: office
        ports:
        - containerPort: 8082
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent      
      volumes: 
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-office
  name: acme-financial-back-office
spec:
  ports:
  - name: http
    port: 8082
    protocol: TCP
    targetPort: 8082
  selector:
    app: acme-financial-office
  sessionAffinity: None
  type: ClusterIP

4.3 account服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-account
  name: acme-financial-account
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: acme-financial-account
  template:
    metadata:
      labels:
        app: acme-financial-account
    spec:
      initContainers:
      - image: registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-account"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.168.30.68:22335"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-account:v0.1
        imagePullPolicy: Always
        name: account
        ports:
        - containerPort: 8083

          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent      
      volumes: 
      - name: sw-agent
        emptyDir: {}
---

apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-account
  name: acme-financial-account
spec:
  ports:
  - name: http
    port: 8083
    protocol: TCP
    targetPort: 8083
  selector:
    app: acme-financial-account
  sessionAffinity: None
  type: ClusterIP

4.4 customer服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: acme-financial-customer
  name: acme-financial-customer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: acme-financial-customer
  template:
    metadata:
      labels:
        app: acme-financial-customer
    spec:
      initContainers:
      - image: registry.cn-shenzhen.aliyuncs.com/gzlj/sw-agent:8.0.1
        name: skywalking-sidecar
        command: ["sh"]
        args: [
                "-c",
                "mkdir -p /opt/sw/agent && cp -rf /opt/skywalking/agent/* /opt/sw/agent/"
        ]
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent
      containers:
      - env:
        - name: JAVA_OPTS
          value: "-javaagent:/opt/sw/agent/skywalking-agent.jar"
        - name: SW_AGENT_NAME
          value: "acme-financial-customer"
        - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
          value: "192.168.30.68:22335"
        image: registry.cn-shenzhen.aliyuncs.com/gzlj/acme-financial-customer:v0.1
        imagePullPolicy: Always
        name: customer
        ports:
        - containerPort: 8084
          protocol: TCP
        volumeMounts:
        - name: sw-agent
          mountPath: /opt/sw/agent      
      volumes: 
      - name: sw-agent
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: acme-financial-customer
  name: acme-financial-customer
  namespace: default
spec:
  ports:
  - name: http
    port: 8084
    protocol: TCP
    targetPort: 8084
  selector:
    app: acme-financial-customer
  sessionAffinity: None
  type: ClusterIP

4.5 业务微服务部署结果

部署业务服务成功，如图所示，UI服务的NodePort为30115。

5 访问springboot业务微服务并查看skywalking

5.1 访问UI服务的三个接口

通过NodePort访问UI服务的三个接口：/hello、/start、/readtimeout。

5.2 访问skywalking的UI界面

打开skywalking的界面（192.168.39.33:16622），即可看见链路追踪相关的数据。