Deploy and use KubeEdge in Kubernetes

Author: Ma Wei, container consultant of Qingyun Technology, cloud-native enthusiast, currently focusing on cloud-native technology, and the technology stack in the cloud-native field involves Kubernetes, KubeSphere, KubeKey, etc.

Edge computing is widely used in manufacturing, industrial, retail, and financial industries. With the rise of cloud-native applications, features such as immutable infrastructure and fast application delivery are very suitable for edge computing scenarios. Therefore, the use of edge computing frameworks on Kubernetes is a very hot direction in recent years. This article will introduce the scenarios and architecture of edge computing, and use a Demo example to show how to run an edge application on an edge node.

Edge Computing Pain Points and Scenarios

First of all, edge computing is an extension of cloud computing. The on-demand and resource pooling characteristics of cloud computing can meet the needs of improving resource utilization and centralized supply of computing resources. However, the application scenario of edge testing determines that it is impossible to throw all applications in the data center. inside. for example

• Low latency processing. In the Internet of Vehicles scenario, if data sharing and command issuance are to be performed, extremely low-latency communication guarantees and computing processing speeds are required.
• Process data locally. Some data is more sensitive and cannot be sent to the cloud (such as user photos, passwords)
• Offline autonomy. Many edge devices do not necessarily have a reliable connection to maintain communication with the cloud. Such as sensors for agriculture and geographic science.

Therefore, for edge nodes and edge devices, unified management and local computing capabilities are required to realize the effect that the cloud is responsible for overall decision-making, and the edge is responsible for local execution. The advantage of this processing is high data processing efficiency, reduced cloud edge bandwidth, and reduced operating costs, but it does not lack cloud asset management, scheduling monitoring, and security control.

After understanding the meaning of edge computing, I have a question. Since edge computing and cloud computing are both concepts, there are bound to be many edge computing products and standards on the market. Why use edge computing on Kubernetes?

My personal understanding is that it goes both ways. Kubernetes itself is more suitable for infrastructure standardization and application delivery. It can help the cloud to manage edge nodes and edge devices in a unified way. It is also suitable for cloud distribution of standard applications. It is an open-source and open-source observable system with rich ecology. It is also suitable for different enterprises to achieve edge-end observability by managing data centers. And Kubernetes also needs edge computing to extend its capabilities more to achieve more platform capabilities and platform standards. After all, cloud-network-edge-end is every direction that contemporary cloud computing needs to cover~

Common Edge Computing Frameworks

For product selection, the most convenient thing in the cloud native field is to open landscape. Let's take a look at the Automation&Configuration column:

There are three common ones, such as KubeEdge, OpenYurt and SuperEdge. In this article, I will first share the KubeEdge incubation project.

KubeEdge architecture

KubeEdge is divided into cloud and edge. The cloud core component CloudCore and the edge-end core component EdgeCore jointly implement many functions of the edge computing framework, such as cloud-edge communication, device management, and offline autonomy. There are also some auxiliary components such as EdgeMesh for edge-end communication and service governance, and Sedna for edge-end AI framework.

As for the specific composition of CloudCore and EdgeCore, you can learn the architecture design in detail from the following figure:

the cloud

CloudCore consists of CloudHub, EdgeController, and DeviceController.

• CloudHub. It mainly observes changes in the cloud, reads and writes Edge messages, caches data and sends it to EdgeHub through WebSocket/QUIC (the listwatch mechanism of K8s is too resource-intensive), and sends some messages obtained through communication with Edge to the Controller.
• EdgeController. As a bridge between ApiServer and EdgeCore, it manages common configuration, Pod, cache and other events, and synchronizes the status of many event information of Pod subscribed by EdgeCore to ApiServer. It also synchronizes events such as ADD/UPDATE/DELETE of ApiServer to EdgeCore.
• DeviceController. Synchronize device updates through EdgeCore DeviceTwin, the overall process is Mapper—>MQTT—>EventBus—>DeviceTwin—>EdgeHub—>CloudHub—>Deviceontroller—>APIServer. On the other hand, the device created in the cloud is sent to the edge to obtain metadata for device update.

edge

• EdgeHub. Cloud-side communication and side-end synchronization of resource updates.
• EventBus. Send/receive MQTT messages
• MetaManager. Store data in SQLlite, which is the message processor of Edged and EdgeHub.
• Edged. Edge-cut version of kubelet. Manage the life cycle of Pod, configmap, volume and other resources. Also contains a StatusManager and MetaClient components. The former uploads the status information stored in the local database to the cloud every 10 seconds, and the latter acts as a client to interact with the local mini Etcd (MetaManager), such as reading the ConfigMap and Secret sent by the cloud, and writing Node and Pod Status.
• DeviceTwin. Store device properties and status, create Edge device and node relationships, and synchronize device properties to the cloud.
• ServiceBus: Receive service requests on the cloud and interact with edge applications through http

Installation and deployment

Install Cloudcore

KubeSphere has integrated KubeEdge, which can provide functions such as edge node management, application delivery, and log monitoring. Next, the edge computing Demo will be demonstrated on KubeSphere.

Enable KubeEdge on KubeSphere, edit cluster configuration, set edgeruntime enabled to true, kubeedge enabled to true, and set advertiseAddress IP to "master_ip"

After the setting is complete, "Edge Node" will appear in Cluster Management -> Node:

At this point, check the workload and configuration under the kubeedge namespace to get familiar with the deployment architecture.

CloudCore runs as a Deployment, and IptablesManager will help process the delivery of Iptables rules for the cloud-side channel.

View the ConfigMap and Secret mounted by CloudCore. ConfigMap mainly mounts the cloudcore.yaml configuration file to /etc/kubeedge/config, which can flexibly modify the configuration of CloudCore Modules. Secret mounts some TLS certificates needed by CloudCore and EdgeCore.

Add edge node

Enter KubeSphere Console, navigate to Node -> Edge Node, and add an edge node:

Fill in the name of the edge node and the IP of the edge node, generate the edge node configuration command, paste it to the edge node and execute:

Since the cloudcore service we use is exposed through NodePort, when registering Cloudcore at the edge node, you need to use the form NodeIP:NodePort, here replace 10000-10004 with 30000-30004 port.

install MQTT service successfully.
kubeedge-v1.9.2-linux-amd64.tar.gz checksum:
checksum_kubeedge-v1.9.2-linux-amd64.tar.gz.txt content:
[Run as service] start to download service file for edgecore
[Run as service] success to download service file for edgecore
kubeedge-v1.9.2-linux-amd64/
kubeedge-v1.9.2-linux-amd64/edge/
kubeedge-v1.9.2-linux-amd64/edge/edgecore
kubeedge-v1.9.2-linux-amd64/version
kubeedge-v1.9.2-linux-amd64/cloud/
kubeedge-v1.9.2-linux-amd64/cloud/csidriver/
kubeedge-v1.9.2-linux-amd64/cloud/csidriver/csidriver
kubeedge-v1.9.2-linux-amd64/cloud/admission/
kubeedge-v1.9.2-linux-amd64/cloud/admission/admission
kubeedge-v1.9.2-linux-amd64/cloud/cloudcore/
kubeedge-v1.9.2-linux-amd64/cloud/cloudcore/cloudcore
kubeedge-v1.9.2-linux-amd64/cloud/iptablesmanager/
kubeedge-v1.9.2-linux-amd64/cloud/iptablesmanager/iptablesmanager
KubeEdge edgecore is running, For logs visit: journalctl -u edgecore.service -b

Check the edge nodes of the KubeSphere console, and you can already see that the edge nodes are registered:

Use kubectl to view the node status:

Metrics & Logs

At this time, we found that the CPU memory information of the node cannot be counted. We need to open KubeSphere Metrics_Server and open EdgeStream on the Edge side:

Edit cc, open metrics-server:

Edit the edge node /etc/kubeedge/config/edgecore.yaml file, search for edgeStream, and change false to true:

  edgeStream:
    enable: true
    handshakeTimeout: 30
    readDeadline: 15
    server: 192.168.100.7:30004
    tlsTunnelCAFile: /etc/kubeedge/ca/rootCA.crt
    tlsTunnelCertFile: /etc/kubeedge/certs/server.crt
    tlsTunnelPrivateKeyFile: /etc/kubeedge/certs/server.key
    writeDeadline: 15

Here the server field sets the port to 30004, because we use the NodePort port to communicate with the cloud

Restart edgecore.service:

$ systemctl restart edgecore.service

View node monitoring information:

In the above chapters, we have enabled edgestream on edgecore to realize the function of collecting metrics of edge nodes in the cloud. This operation also realizes the ability to view edge logs.

Generally speaking, when we run kubectl logs pod -n namespace, kubectl will request kube-apiserver to check whether the pod exists and whether there are multiple containers in the pod, and then retrieve the Kubelet Server information of the Node where the Pod is located. This information can generally be found through kubectl describe or get node:

$ kubectl get node ks2 -oyaml
  addresses:
  - address: 192.168.100.7
    type: InternalIP
  - address: ks2
    type: Hostname
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
kubectl get node edge-node-1 -oayml
  addresses:
  - address: 192.168.100.6
    type: InternalIP
  - address: edge-node-1
    type: Hostname
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10352

InternalIP+kubeletEndpoint constitutes the address of the kubelet server, and kubectl logs can request this address to obtain relevant log information. But for the edge end, the internalIP cloud is inaccessible in most cases.

At this time, CloudCore's CloudStream and EdgeCore's EdgeStream need to establish a cloud-side channel. After CloudCore and EdgeCore establish cloud-side WebSocket communication, the requested Edge kubelet server log information can be returned to the cloud through the channel. This channel requires CloudStream and EdgeStream to be opened on both sides, and verified by TLS certificates.

EdgeStream has been enabled on the edge side, and it will be automatically enabled after CloudCore is deployed on the cloud.

View the CloudCore.yaml configuration file mounted on the cloud:

Of course, the cloud edge channel can only return logs from messages, specifically to CloudCore’s CloudStream, and a NAT rule needs to be set:

$ iptables -t nat -A OUTPUT -p tcp --dport 10350（edge kubelet 端口）-j NAT --to $ClOUDCOREIPS:10003

This operation has been completed by the automatically deployed iptables-manager~

$ iptables -t nat -L
Chain TUNNEL-PORT (2 references)
target     prot opt source               destination
DNAT       tcp  --  anywhere             anywhere             tcp dpt:10351 to:10.20.253.88:10003
DNAT       tcp  --  anywhere             anywhere             tcp dpt:10352 to:10.20.253.127:10003

Enter the edge node and check the container group. We can see that there are several daemonsets with strong tolerance and dispatched to the edge node. Due to the unstable communication in many cases at the edge end, it is not suitable for running CNI components such as Calico. EdgeMesh is more used for For cloud-side communication and service discovery, we can manually Patch Pods to prevent non-edge nodes from being scheduled to work nodes:

#!/bin/bash
NoShedulePatchJson='{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"node-role.kubernetes.io/edge","operator":"DoesNotExist"}]}]}}}}}}}'
ns="kube-system"
DaemonSets=("nodelocaldns" "kube-proxy" "calico-node")
length=${#DaemonSets[@]}
for((i=0;i<length;i++));  
do
         ds=${DaemonSets[$i]}
        echo "Patching resources:DaemonSet/${ds}" in ns:"$ns",
        kubectl -n $ns patch DaemonSet/${ds} --type merge --patch "$NoShedulePatchJson"
        sleep 1
done

Enter the node terminal (available above KS3.3), run the script

sh-4.2# ./bash.sh
Patching resources:DaemonSet/nodelocaldns in ns:kube-system,
daemonset.apps/nodelocaldns patched
Patching resources:DaemonSet/kube-proxy in ns:kube-system,
daemonset.apps/kube-proxy patched
Patching resources:DaemonSet/calico-node in ns:kube-system,
daemonset.apps/calico-node patched

View edge node container groups:

run the application

Now that the edge node is registered, let's run an application.

Go to Project - Application Load and create a Deployment workload:

Set resource limits

Select node assignment:

After creation, it will be displayed that the node has taints and cannot be scheduled, and toleration needs to be added:

Edit the yaml of the nginx-edge application and add the toleration for edge:

After adding the taint tolerance, the Pod runs successfully:

Set up a NodePort Service to access the Nginx service:

At this time, it can be found that the access is unavailable, because the cloud cannot access the Pod network at the edge, check the ip of nginx-edge at the edge:

bash-5.1# kubectl get pod -n demo -o wide
NAME                          READY   STATUS    RESTARTS      AGE     IP              NODE          NOMINATED NODE   READINESS GATES
nginx-9c99b5774-vvsfq         1/1     Running   4 (12h ago)   4d11h   10.20.253.123   ks2           <none>           <none>
nginx-edge-68c66d6bf9-k9l6n   1/1     Running   0             7m55s   172.17.0.2      edge-node-1   <none>           <none>

ssh to the edge node, access 172.17.0.2:

$ curl 172.17.0.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

It can be seen that the Nginx Pod service is normal, and the IP address allocation is also very familiar, 172.17.0.0/16, this is not the docker bridge network segment, check the edge node docker0 address:

$ ip ad
3: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:1a:fa:b2:cb brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:1aff:fefa:b2cb/64 scope link
       valid_lft forever preferred_lft forever

$ docker ps
CONTAINER ID   IMAGE                COMMAND                  CREATED          STATUS          PORTS     NAMES
b6c296a4dc16   nginx                "/docker-entrypoint.…"   7 minutes ago    Up 6 minutes              k8s_container-zpnv7r                           _nginx-edge-68c66d6bf9-k9l6n_demo_99d01b2c-9ca7-4d56-a475-ab1b6838a35d_0
4e72f538703c   kubeedge/pause:3.1   "/pause"                 11 minutes ago   Up 11 minutes             k8s_POD_nginx-edge-6                           8c66d6bf9-k9l6n_demo_99d01b2c-9ca7-4d56-a475-ab1b6838a35d_0

We can see that the nginx-edge container still runs in the form of a Pod, sharing the network namespace through the pause container, but it does not use the cluster's PodCIDR allocation, but uses the docker bridge network. Therefore, without the service discovery of coredns and kube-proxy and the intercommunication of the cloud-side container network, the edge-end Pod Service cannot be accessed. This is in line with the characteristics of the edge cloud. The edge end is more of a private network and cannot be accessed by the cloud. This one-way communication requires other forms of network to facilitate cloud-edge communication and service access, such as establishing tunnels.

Cloud-side service mutual access

There is an EdgeMesh project in the KubeEdge community. In the scenario of edge computers, the network topology is more complex. Edge nodes in different regions are often not interconnected, and applications need to communicate with each other. EdgeMesh can meet the requirements of traffic communication between edge nodes. According to the official Github introduction, EdgeMesh, as the data plane component of the KubeEdge cluster, provides simple service discovery and traffic proxy functions for applications, thus shielding the complex network structure in edge scenarios.

Therefore, EdgeMesh mainly achieves two ultimate goals:

• Users can access edge-to-edge, edge-to-cloud, and cloud-to-edge applications in different networks
• Deploying EdgeMesh is equivalent to deploying CoreDNS+Kube-Proxy+CNI

Deploy Edge Mesh

There are some prerequisites for deployment, such as enabling Edge Kube-API Endpoint: enabling cloudcore dynamicController:

$ vim /etc/kubeedge/config/cloudcore.yaml
modules:
  ...
  dynamicController:
    enable: true

Start Edge metaServer:

$ vim /etc/kubeedge/config/edgecore.yaml
modules:
  ...
  edgeMesh:
    enable: false
  ...
  metaManager:
    metaServer:
      enable: true

Add edgemesh commonconfig information:

$ vim /etc/kubeedge/config/edgecore.yaml
modules:
  ...
  edged:
    clusterDNS: 169.254.96.16
    clusterDomain: cluster.local
...

After restarting cloudcore and edgecore, you can verify whether the kube-API can be requested on the edge side:

[root@i-pfcctw6w ~]#  curl 127.0.0.1:10550/api/v1/services
{"apiVersion":"v1","items":[{"apiVersion":"v1","kind":"Service","metadata":{"creationTimestamp":"2023-01-04T13:09:51Z","labe                           ls":{"component":"apiserver","provider":"kubernetes","service.edgemesh.kubeedge.io/service-proxy-name":""}······

EdgeMesh Helm Chart has been included in the KubeSphere application store, we can open the application store and deploy it directly. Enter the kubeedge project and deploy EdgeMesh to this project. At this time, you need to modify the server.nodeName and server.advertiseAddress of the application settings. Execute the installation, and check the running status of edgemesh-server and edgemesh-agent after the installation is successful:

Test side-end service access:

Using the example application https://github.com/kubeedge/edgemesh/examples, deploy a hostname application and service: Test accessing the edge Service from the Pod in the cloud:

Test cloud-side mutual access:

Deploy cloudzone and edgezone applications of https://github.com/kubeedge/edgemesh/examples : cloud busybox access edge application: edge busybox access cloud application:

$ docker exec -it 5a94e3e34adb sh
/ # cat /etc/resolv.conf
nameserver 169.254.96.16
search edgezone.svc.ks2 svc.ks2 ks2 sh1.qingcloud.com
options ndots:5
/ # telnet tcp-echo-cloud-svc.cloudzone 2701
Welcome, you are connected to node ks2.
Running on Pod tcp-echo-cloud-6d687d88c4-tllst.
In namespace cloudzone.
With IP address 10.20.253.177.
Service default.

Edge device data access

Deploy an App that simulates temperature data acquisition:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: temperature-mapper
  labels:
    app: temperature
spec:
  replicas: 1
  selector:
    matchLabels:
      app: temperature
  template:
    metadata:
      labels:
        app: temperature
    spec:
      hostNetwork: true
      tolerations:
      - key: "node-role.kubernetes.io/edge"
        operator: "Exists"
        effect: "NoSchedule"
      nodeSelector:
        kubernetes.io/hostname: "edge-node-1"
      containers:
      - name: temperature
        image: lammw12/temperature-mapper:edge
        imagePullPolicy: IfNotPresent
        securityContext:
          privileged: true

Create a DeviceModel:

apiVersion: devices.kubeedge.io/v1alpha2
kind: Device
metadata:
  name: temperature
  labels:
    description: 'temperature'
    manufacturer: 'test'
spec:
  deviceModelRef:
    name: temperature-model
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
          - key: ''
            operator: In
            values:
              - edge-centos
status:
  twins:
    - propertyName: temperature-status
      desired:
        metadata:
          type: string
        value: ''

Create Device:

apiVersion: devices.kubeedge.io/v1alpha2
kind: Device
metadata:
  name: temperature
  labels:
    description: 'temperature'
    manufacturer: 'test'
spec:
  deviceModelRef:
    name: temperature-model
  nodeSelector:
    nodeSelectorTerms:
      - matchExpressions:
          - key: ''
            operator: In
            values:
              - edge-node-1
status:
  twins:
    - propertyName: temperature-status
      desired:
        metadata:
          type: string
        value: ''

KubeEdge adds two resources, DeviceModel and Device, through the CRD of Kubernetes to describe device meta-information and device instance information respectively. DeviceController is responsible for edge device management and transfers this information between the cloud and the edge. Users can create, update, and delete device metadata from the cloud through the Kubernetes API, and can also control the desired state of device attributes through the CRD API, and perform CRUD operations on the device from the cloud. DeviceModel describes device properties, such as "temperature" or "pressure", and acts like a reusable template with which many devices can be created and managed. A Device instance represents an actual device object. It's like an instantiation of the device model, referencing the properties defined in the model. kubectl apply the above resources. View the running temperature application: View the temperature application log: Use kubectl to view the device status:

[root@ks2 examples]# kubectl get device temperature -oyaml
···
status:
  twins:
  - desired:
      metadata:
        type: string
      value: ""
    propertyName: temperature-status
    reported:
      metadata:
        timestamp: "1673256318955"
        type: string
      value: 70C

The device status in yaml contains two pieces of data, one is the status data that the cloud wants to set ('desired'), and the other is the status data reported by the edge ('reported'). The DeviceController in the cloud monitors the creation event of the device through the Kubernetes API, and automatically creates a new configmap, stores the status and other attribute information of the device, and saves it in ectd. EdgeController synchronizes the configmap to the edge node, so the application of the edge node can also obtain the attribute information of the device. The 'desired' value will be initialized to the edge node database and edge device, so even if the edge node restarts, it can automatically restore to the previous state. Of course, this 'desired' value will also change as the cloud user CRUDs the device.

Mirror warm-up

In practical applications, the edge nodes and devices are large-scale and the network environment is not stable enough. After the edge application is delivered to multiple nodes in the cloud, the time it takes to pull the image may be very troublesome for the administrator. This runs counter to the expectation of containerized fast and large-scale delivery of applications and business rollout/expansion/upgrade. Therefore, the image warm-up capability is indispensable in large-scale edge node scenarios. We can use the image warm-up tool to pull images on edge nodes in advance to speed up application deployment. There is an OpenKruise project in the open source community, which can realize this requirement. OpenKruise hosts a Daemonset for each Node, and bypasses kubelet to achieve the ability to pull images by interacting with CRI. For example, define a NodeImage CR, define what image each node needs to warm up, and then kruise-daemon can perform image pulling tasks according to the NodeImage: for large-scale edge node scenarios, you can use ImagePullJob to filter nodes and perform batch warmup , after an ImagePullJob is created, it will be received and processed by the imagepulljob-controller in kruise-manager, and it will be decomposed and written to the NodeImages of all matching nodes, so as to complete the scale warm-up.

apiVersion: apps.kruise.io/v1alpha1
kind: ImagePullJob
metadata:
  name: job-with-always
spec:
  image: nginx:1.9.1   # [required] 完整的镜像名 name:tag
  parallelism: 10      # [optional] 最大并发拉取的节点梳理, 默认为 1
  selector:            # [optional] 指定节点的 名字列表 或 标签选择器 (只能设置其中一种)
    names:
    - node-1
    - node-2
    matchLabels:
      node-type: xxx
# podSelector:         # [optional] 通过 podSelector 匹配Pod，在这些 Pod 所在节点上拉取镜像, 与 selector 不能同时设置.
#   matchLabels:
#     pod-label: xxx
#   matchExpressions:
#   - key: pod-label
#      operator: In
#        values:
#        - xxx
  completionPolicy:
    type: Always                  # [optional] 默认为 Always
    activeDeadlineSeconds: 1200   # [optional] 无默认值, 只对 Alway 类型生效
    ttlSecondsAfterFinished: 300  # [optional] 无默认值, 只对 Alway 类型生效
  pullPolicy:                     # [optional] 默认 backoffLimit=3, timeoutSeconds=600
    backoffLimit: 3
    timeoutSeconds: 300

The above is the whole content of this sharing. The main goal is to use the edge computing of KubeSphere, so to what extent this sword dance, we must continue to polish technology + professional services.

This article is published by OpenWrite , a multi- post platform for blogging!

{{o.name}}

{{m.name}}