Preface
Superedge is a Kubernetes-native edge computing management framework launched by Tencent. Compared with openyurt and kubeedge , superedge not only has the features of Kubernetes zero intrusion and edge autonomy, but also supports advanced features such as unique distributed health checks and edge service access control, which greatly reduces the impact of cloud edge network instability on services. It also greatly facilitates the release and governance of edge cluster services
characteristic
- Kubernetes-native : Superedge is expanded on the basis of native Kubernetes, adding certain components of edge computing, which is completely non-invasive to Kubernetes; in addition, by simply deploying superedge core components, the native Kubernetes cluster can enable the edge computing function; in addition, zero intrusion Make it possible to deploy any Kubernetes native workload (deployment, statefulset, daemonset, and etc) on the edge cluster
- Edge autonomy : Superedge provides L3 level edge autonomy. When the edge node is unstable or disconnected from the cloud network, the edge node can still operate normally without affecting the edge services already deployed
- Distributed health check : Superedge provides side-end distributed health check capabilities. Each edge node will deploy edge-health. Edge nodes in the same edge cluster will perform health checks on each other and vote on the status of the nodes. In this way, even if there is a problem with the cloud edge network, as long as the connection between the edge nodes is normal, the node will not be evicted; in addition, the distributed health check also supports grouping, dividing the cluster nodes into multiple groups (nodes in the same computer room). The nodes in each group check each other. The advantage of this is to avoid the data interaction between the nodes after the cluster increases, and it is difficult to reach agreement; at the same time, it also adapts to the edge nodes in the network The topology is naturally grouped. The entire design avoids a large number of pod migration and reconstruction caused by the instability of the cloud side network, and ensures the stability of the service
- Service access control : Superedge self-developed ServiceGroup to implement service access control based on edge computing. Based on this feature, only two custom resources, DeploymentGrid and ServiceGrid, can be built, and a set of services can be deployed in different computer rooms or regions that share the same cluster conveniently, and the requests between each service can be in the local computer room or local domain. It can be completed (closed loop), avoiding cross-regional access of services. Using this feature can greatly facilitate the release and governance of edge cluster services
- Cloud Edge Tunnel : Superedge supports self-built tunnels (currently supports TCP, HTTP and HTTPS) to solve cloud edge connection problems in different network environments. Realize the unified operation and maintenance of IP edge nodes without public network
Overall structure
The component functions are summarized as follows:
Cloud component
In addition to the native Kubernetes master components (cloud-kube-apiserver, cloud-kube-controller and cloud-kube-scheduler) deployed in the edge cluster, the main control components of the cloud include:
- tunnel-cloud : responsible for maintainingthe network tunnelwith the edge node tunnel-edge , currently supports TCP/HTTP/HTTPS protocols
- application-grid controller : The Kubernetes Controller corresponding to the service access control ServiceGroup is responsible for managing DeploymentGrids and ServiceGrids CRDs, and the corresponding Kubernetes deployment and service are generated by these two CRs. At the same time, the self-developed realization of service topology awareness enables closed-loop service access
- edge-admission : Determine whether the node is healthy through the status report of the distributed health check of the edge node, and assist the cloud-kube-controller to perform related processing actions (taint)
Edge component
In addition to the kubelet and kube-proxy that need to be deployed on the native Kubernetes worker node, the edge side also adds the following edge computing components:
- lite-apiserver : The core component of edge autonomy is the proxy service of cloud-kube-apiserver, which caches certain requests of edge node components to apiserver. When these requests are encountered and there are problems with the cloud-kube-apiserver network, it will Return directly to the client
- edge-health : Edge-end distributed health check service, responsible for performing specific monitoring and detection operations, and voting to determine whether the node is healthy
- tunnel-edge : Responsible for establishinga network tunnelwith the cloud edge cluster tunnel-cloud , accepting API requests, and forwarding them to the edge node component (kubelet)
- application-grid wrapper : combined with application-grid controller to complete closed-loop service access in ServiceGrid (service topology awareness)
Functional Overview
Application deployment & service access control
Superedge can support the application deployment of all workloads of native Kubernetes, including:
- deployment
- statefulset
- daemonset
- job
- cronjob
For edge computing applications, it has the following unique points:
- In edge computing scenarios, multiple edge sites are often managed in the same cluster, and each edge site has one or more computing nodes
- At the same time, I hope to run a set of services with business logic in each site. The services in each site are a complete set of functions that can provide users with services
- Due to network restrictions, cross-site access between services with business connections is undesirable or impossible
In order to solve the above problems, superedge innovatively constructed the ServiceGroup concept, which is convenient for users to conveniently deploy a set of services in different computer rooms or areas that share the same cluster, and make the requests between each service in the local computer room or local domain. It can be completed (closed loop), avoiding cross-regional service access
Several key concepts are involved in ServiceGroup:
NodeUnit
- NodeUnit is usually one or more computing resource instances located in the same edge site, and it is necessary to ensure that the intranet of the nodes in the same NodeUnit is connected
- Services in the ServiceGroup group run in a NodeUnit
- ServiceGroup allows users to set the number of pods (belongs to deployment) the service runs in a NodeUnit
- ServiceGroup can restrict calls between services to this NodeUnit
NodeGroup
- NodeGroup contains one or more NodeUnit
- Ensure that the services in the ServiceGroup are deployed on each NodeUnit in the collection
- When a NodeUnit is added to the cluster, the services in the ServiceGroup will be automatically deployed to the new NodeUnit
ServiceGroup
- ServiceGroup contains one or more business services
- Applicable scene:
- Business needs to be packaged and deployed;
- Need to run in each NodeUnit and guarantee the number of pods
- The calls between services need to be controlled in the same NodeUnit, and traffic cannot be forwarded to other NodeUnits
- Note: ServiceGroup is an abstract resource concept, multiple ServiceGroups can be created in a cluster
Here is a specific example to illustrate the ServiceGroup function:
# step1: labels edge nodes
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
node0 Ready <none> 16d v1.16.7
node1 Ready <none> 16d v1.16.7
node2 Ready <none> 16d v1.16.7
# nodeunit1(nodegroup and servicegroup zone1)
$ kubectl --kubeconfig config label nodes node0 zone1=nodeunit1
# nodeunit2(nodegroup and servicegroup zone1)
$ kubectl --kubeconfig config label nodes node1 zone1=nodeunit2
$ kubectl --kubeconfig config label nodes node2 zone1=nodeunit2
# step2: deploy echo DeploymentGrid
$ cat <<EOF | kubectl --kubeconfig config apply -f -
apiVersion: superedge.io/v1
kind: DeploymentGrid
metadata:
name: deploymentgrid-demo
namespace: default
spec:
gridUniqKey: zone1
template:
replicas: 2
selector:
matchLabels:
appGrid: echo
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
appGrid: echo
spec:
containers:
- image: gcr.io/kubernetes-e2e-test-images/echoserver:2.2
name: echo
ports:
- containerPort: 8080
protocol: TCP
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
resources: {}
EOF
deploymentgrid.superedge.io/deploymentgrid-demo created
# note that there are two deployments generated and deployed into both nodeunit1 and nodeunit2
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
deploymentgrid-demo-nodeunit1 2/2 2 2 5m50s
deploymentgrid-demo-nodeunit2 2/2 2 2 5m50s
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
deploymentgrid-demo-nodeunit1-65bbb7c6bb-6lcmt 1/1 Running 0 5m34s 172.16.0.16 node0 <none> <none>
deploymentgrid-demo-nodeunit1-65bbb7c6bb-hvmlg 1/1 Running 0 6m10s 172.16.0.15 node0 <none> <none>
deploymentgrid-demo-nodeunit2-56dd647d7-fh2bm 1/1 Running 0 5m34s 172.16.1.12 node1 <none> <none>
deploymentgrid-demo-nodeunit2-56dd647d7-gb2j8 1/1 Running 0 6m10s 172.16.2.9 node2 <none> <none>
# step3: deploy echo ServiceGrid
$ cat <<EOF | kubectl --kubeconfig config apply -f -
apiVersion: superedge.io/v1
kind: ServiceGrid
metadata:
name: servicegrid-demo
namespace: default
spec:
gridUniqKey: zone1
template:
selector:
appGrid: echo
ports:
- protocol: TCP
port: 80
targetPort: 8080
EOF
servicegrid.superedge.io/servicegrid-demo created
# note that there is only one relevant service generated
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 192.168.0.1 <none> 443/TCP 16d
servicegrid-demo-svc ClusterIP 192.168.6.139 <none> 80/TCP 10m
# step4: access servicegrid-demo-svc(service topology and closed-looped)
# execute on onde0
$ curl 192.168.6.139|grep "node name"
node name: node0
# execute on node1 and node2
$ curl 192.168.6.139|grep "node name"
node name: node2
$ curl 192.168.6.139|grep "node name"
node name: node1The above example summarizes ServiceGroup as follows:
- NodeUnit, NodeGroup, and ServiceGroup are all a concept. Specifically, the corresponding relationship in actual use is as follows:
- NodeUnit is a group of edge nodes with the same label key and value
- NodeGroup is a group of NodeUnits (different values) with the same label key
- ServiceGroup is specifically composed of two CRDs: DepolymentGrid and ServiceGrid, with the same gridUniqKey
- The gridUniqKey value corresponds to the label key of the NodeGroup, that is, the ServiceGroup corresponds to the NodeGroup one-to-one, and the NodeGroup corresponds to multiple NodeUnits. At the same time, each NodeUnit in the NodeGroup will deploy the ServiceGroup corresponding deployment. These deployments (named deploymentgridName-NodeUnit) pass the nodeSelector. Harmony fixes on a NodeUnit and restricts access within the NodeUnit through service topology awareness
Distributed health check
In the edge computing scenario, the network environment between the edge node and the cloud is very complex, and the connection is not reliable. In the native Kubernetes cluster, the connection between the apiserver and the node will be interrupted, the node status will be abnormal, and the pod will be expelled and the endpoint will be missing. Cause service interruption and fluctuation, specifically the native Kubernetes processing is as follows:
- The disconnected node is set to ConditionUnknown state, and NoSchedule and NoExecute taints are added
- The pod on the lost node is expelled and rebuilt on other nodes
- The pod on the lost node is removed from the Endpoint list of the Service
Therefore, the edge computing scenario only relies on the connection between the edge and the apiserver is not enough to determine whether the node is abnormal, and it will cause misjudgment due to the unreliability of the network and affect normal services. Compared with the connection between the cloud and the edge, it is obvious that the connection between the edge nodes is more stable and has certain reference value, so superedge proposes an edge distributed health check mechanism. In addition to the factors of apiserver, the node status judgment in this mechanism also introduces node evaluation factors to make a more comprehensive status judgment of the node. Through this function, a large number of pod migration and reconstruction caused by the unreliable cloud side network can be avoided, and the stability of the service can be ensured
Specifically, the accuracy of node status judgment is mainly enhanced through the following three levels:
- Each node periodically detects the health status of other nodes
- All nodes in the cluster regularly vote to determine the status of each node
- The cloud and the edge node jointly determine the node state
The final judgment processing of the distributed health check is as follows:
| The final state of the node | Cloud judged to be normal | Cloud judgment abnormal |
|---|---|---|
| Node internal judgment is normal | normal | No more scheduling new pods to this node (NoSchedule taint) |
| Node internal judgment abnormal | normal | Evict the stock pod; remove the pod from the Endpoint list; no longer schedule new pods to the node |
Marginal autonomy
For edge computing users, in addition to enjoying the convenience of management and operation and maintenance brought by Kubernetes itself, they also want to have disaster tolerance in a weak network environment. Specifically, they are as follows:
- Even if the node loses connection with the master, the business on the node can continue to run
- Ensure that if the business container exits abnormally or hangs up, kubelet can continue to pull up
- Also ensure that after the node restarts , the business can continue to be pulled up again
- Users deployed in the plant is a micro-services , need to ensure that node after the restart, with a micro-plant service can access
For the standard Kubernetes, if the node is disconnected from the network and restarts abnormally, the symptoms are as follows:
- The state of the lost node is set to the ConditionUnknown state
- After the business process on the lost node exits abnormally, the container can be pulled up
- The Pod IP on the lost node is removed from the Endpoint list
- After the missing node is restarted, all the containers will disappear without being pulled up
The self-developed edge autonomy of superedge is to solve the above problems. Specifically, edge autonomy can achieve the following effects:
- The node will be placed in the ConditionUnknown state, but the service is still available (pod will not be evicted and removed from the endpoint list)
- In the case of multi-node network disconnection, the Pod business runs normally and the microservice capabilities are provided normally
- After multiple nodes are disconnected from the network and restarted, the Pod will be pulled up again and run normally
- After multiple nodes are disconnected from the network and restarted, all microservices can be accessed normally
Among them, the first two points can be achieved through the distributed health check mechanism introduced above, and the following two points can be achieved through lite-apiserver, network snapshot and DNS solutions, as follows:
lite-apiserver mechanism
Superedge adds a layer of mirroring lite-apiserver component on the edge, so that all edge nodes' requests for cloud kube-apiserver will point to the lite-apiserver component:
The lite-apiserver is actually a proxy, which caches some kube-apiserver requests, and directly returns to the client when these requests are encountered and cannot communicate with the apiserver:
In general: For edge node components, the function provided by lite-apiserver is kube-apiserver, but on the one hand lite-apiserver is only effective for this node, and on the other hand, it takes up very little resources. When the network is unobstructed, the lite-apiserver component is transparent to the node component; and when the network is abnormal, the lite-apiserver component will return the data required by the node to the component on the node to ensure that the node component will not be affected by the network. Impact of abnormal conditions
Network snapshot
Through lite-apiserver, the pod can be pulled up normally after restarting when the edge node is disconnected from the network, but according to the principle of native Kubernetes, the pod ip after being pulled up will change, which is not allowed in some cases. For this reason Superedge designed a network snapshot mechanism to ensure that the edge node restarts, and the IP is kept unchanged after the pod is pulled up. Specifically, it takes periodic snapshots of the network information of the components on the node and restores it after the node restarts
Local DNS solution
Through the lite-apiserver and network snapshot mechanism, it can be ensured that after the edge node restarts when the network is disconnected, the Pod will be pulled up and run normally, and the microservices are also running normally. The mutual access between services involves a domain name resolution problem: Generally speaking, we use coredns to do domain name resolution within the cluster, and it is generally deployed in the form of Deployment, but in the case of edge computing, nodes may not be the same The local area network is likely to cross-availability zone. At this time, the coredns service may not be accessible. In order to ensure that dns access is always normal, superedge has designed a special local dns solution, as follows:
Local dns uses DaemonSet to deploy coredns to ensure that each node has available coredns. At the same time, modify the startup parameters of kubelet on each node --cluster-dnsto point it to the local private IP (the same for each node). This ensures that domain name resolution can be performed even when the network is disconnected.
In general, superedge is based on the lite-apiserver mechanism, combined with a distributed health check mechanism, network snapshots, and local coredns to ensure the network reliability of the edge container cluster in a weak network environment. In addition, with the higher the level of edge autonomy, more and more components will be required
Cloud side tunnel
Finally, I will introduce the cloud edge tunnel of superedge. The cloud edge tunnel is mainly used to: proxy the request of the cloud to access the edge node components, and solve the problem that the cloud cannot directly access the edge node (the edge node is not exposed to the public network)
The architecture diagram is as follows:
The realization principle is:
- The tunnel-edge on the edge node actively connects to the cloud tunnel-cloud service, and the tunnel-cloud service transfers the request to the specific pod of the tunnel-cloud according to the load balancing policy
- After tunnel-edge and tunnel-cloud establish a grpc connection, tunnel-cloud will write the mapping of its podIp and nodeName of the node where tunnel-edge is located into DNS (tunnel dns). After the grpc connection is disconnected, tunnel-cloud will delete the mapping between the relevant podIp and the node name
The proxy forwarding process of the entire request is as follows:
- When apiserver or other cloud applications access kubelet or other applications on edge nodes, tunnel-dns forwards the request to the pod of tunnel-cloud through DNS hijacking (resolving the node name in the host to podIp of tunnel-cloud)
- tunnel-cloud forwards the request information to the grpc connection established with tunnel-edge corresponding to the node name according to the node name
- tunnel-edge requests the application on the edge node according to the received request information
to sum up
This article in turn introduces the features, overall architecture, main functions and principles of the open source edge computing framework SuperEdge. Among them, distributed health check and edge cluster service access control ServiceGroup are unique features of SuperEdge. Distributed health check largely avoids the migration and reconstruction of a large number of pods caused by the unreliable cloud-side network, and ensures the stability of the service; while the ServiceGroup greatly facilitates users in different computer rooms or regions that belong to the same cluster. Deploy a set of services, and make the requests between each service be completed in the local computer room or in the local domain (closed loop), avoiding cross-regional access of services. In addition, there are functions such as edge autonomy and cloud-side tunnels.
On the whole, SuperEdge adopts a non-intrusive way to build edge clusters. On the basis of the original Kubernetes components remain unchanged, some new components are added to complete edge computing functions. It not only retains the powerful orchestration system of Kubernetes, but also has a complete edge. Calculate ability.