晋晨（OpenYurt）、Vivian Hu（WasmEdge）
Edge computing is a distributed application architecture that places computing resources, such as CPUs and storage, close to data sources. In this way, low latency, high performance and strong security are provided for various applications.
However, similar to cloud computing in data centers, edge computing also faces challenges such as workload distribution, management, and orchestration across heterogeneous devices and networks. To address these challenges in cloud computing above, many developers opt for container-based solutions. As a result, developers are now bringing solutions such as Kubernetes (K8s) and containers to edge computing. Unfortunately, neither K8s nor Linux containers are designed for edge computing.
Potential issues with standard K8s on the edge include:
- Edge devices have limited computing, storage, and network resources. Running a full K8s node on edge devices is often too resource intensive.
- K8s can only manage devices with public IP addresses. However, many edge devices do not expose their public IP.
- Edge devices are often offline, whereas standard K8s requires their pods to be always connected.
- Edge devices are usually connected over non-TCP networks and require dedicated network interfaces to communicate with their K8s manager.
Therefore, developers have created specialized Kubernetes distributions for edge computing. A leading edge-optimized K8s distribution is OpenYurt . Originally created by Alibaba, OpenYurt is now a CNCF sandbox project. OpenYurt is an open platform that includes the powerful orchestration capabilities of upstream Kubernetes. As a result, users can deliver, manage, and monitor edge workloads using the same upstream Kubernetes API.
Even with edge-optimized K8s distributions like OpenYurt , standard Linux containers are still too bulky to run edge workloads on small devices. A typical Linux container image usually takes up space in gigabytes and takes a few seconds to spin up a full CPU. Given that typical edge applications are often lean, Linux containers consume excessive storage, CPU, and battery resources on resource-constrained devices. Additionally, Linux container applications are not cross-platform and insecure on edge devices.
For edge containerized applications, OCI-compliant, k8s-compatible lightweight WebAssembly runtimes, such as WasmEdge , are better alternatives to Linux containers . WasmEdge provides a capability-based security sandbox that makes few assumptions about underlying OS services, while supporting multiple programming languages on the front end . Besides Linux, macOS, Windows, WasmEdge can even run on real-time operating systems (RTOS) like the Linux Foundation's seL4 .
OpenYurt and the WasmEdge team together demonstrate how to use OpenYurt to manage WasmEdge applications in an edge cluster. OpenYurt and WasmEdge will finally bring cloud-native containerized applications to edge computing! let's start!
Ready to work
OpenYurt uses a centralized Kubernetes control plane to manage multiple edge nodes in cloud servers. Additionally, it extends upstream Kubernetes to the edge. In this demo, we will set up two machines. One simulated cloud node is called Master, and the other simulated edge node is called Node. These two nodes form the simplest OpenYurt cluster where OpenYurt components run.
First, close the swap partition
sudo swapoff -a
free -m //验证
Next, configure the files for both nodesnode/etc/hosts
Then, load the kernel module br_netfilter, and adjust the kernel parameters.
//load the module
sudo modprobe br_netfilter
//verify
lsmod | grep br_netfilter
// create k8s.conf
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system
Finally, set the rp-filtervalue , change /etc/sysctl.d/10-network-security.confthe value of the two parameters in the file from 2 to 1, and /proc/sys/net/ipv4/ip_forwardset to 1.
sudo vi /etc/sysctl.d/10-network-security.confecho 1 >
/proc/sys/net/ipv4/ip_forward
sudo sysctl --system
Install containerd
Install containerd on the edge node using the command line below.
export VERSION="1.5.7"
echo -e "Version: $VERSION"
echo -e "Installing libseccomp2 ..."
sudo apt install -y libseccomp2
echo -e "Installing wget"
sudo apt install -y wget
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/cri-containerd-cni-${VERSION}-linux-amd64.tar.gz.sha256sum sha256sum --check cri-containerd-cni-${VERSION}-linux-amd64.tar.gz.sha256sum
sudo tar --no-overwrite-dir -C / -xzf cri-containerd-cni-${VERSION}-linux-amd64.tar.gz
sudo systemctl daemon-reload
Since the crun project supports WasmEdge by default, we only need to configure the containerd configuration for runc. So we need to /etc/containerd/config.tomlmodify the runc parameter in to curn, add pod_annotation.
sudo mkdir -p /etc/containerd/
sudo bash -c "containerd config default > /etc/containerd/config.toml"
wget https://raw.githubusercontent.com/second-state/wasmedge-containers-examples/main/containerd/containerd_config.diff
sudo patch -d/ -p0 < containerd_config.diff
After that, restart containerd for the configuration to take effect.
systemctl start containerd
Install WasmEdge
Install WasmEdge on the edge node using the installation script below.
curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash
install and build crun
We need to build a crun binary that supports WasmEdge on the edge node. Currently, the most straightforward way is to build crun yourself from source. First make sure you have installed the dependencies of crun on Ubuntu 20.04. If you are using another version of Linux, please refer to the official documentation .
sudo apt update
sudo apt install -y make git gcc build-essential pkgconf libtool \
libsystemd-dev libprotobuf-c-dev libcap-dev libseccomp-dev libyajl-dev \
go-md2man libtool autoconf python3 automake
Next, configure, build, install the crunbinaries
sudo make install
git clone https://github.com/containers/crun
cd crun
./autogen.sh
./configure —with-wasmedge
make
sudo make install
Install OpenYurt
We can now install OpenYurt on the Edge node to control the containerd, crun and wasmedge stacks underneath it.
The easiest way to build an OpenYrut cluster is to use the OpenYurt Experience Center. You just need to sign up for an account for testing, and then you can get an OpenYurt cluster. Next, you can use the yurtctl joincommand line to join edge nodes. See more details here . Currently, the OpenYurt component version provided by the OpenYurt Experience Center is 0.6.0 (the latest version), and the Kubernetes version is 1.20.11.
Alternatively, you can build an OpenYurt cluster from scratch by following the guide below.
Set up a K8s cluster
Here we use yurtctl convertto convert the K8s cluster to an OpenYurt cluster, so we need to build a K8s cluster first. One thing to note, if you used yurtctl init/jointo setup an OpenYurt cluster and join nodes, you can skip this step.
$ sudo apt-get update && sudo apt-get install -y ca-certificates curl software-properties-common apt-transport-https
// add K8s source
$ curl -s https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF
$ deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
// install K8s components 1.18.9
$ sudo apt-get update && sudo apt-get install -y kubelet=1.18.9-00 kubeadm=1.18.9-00 kubectl=1.18.9-00
// Initialize the master node
$ sudo kubeadm init --pod-network-cidr 172.16.0.0/16 \
--apiserver-advertise-address=192.168.3.167 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers
// join the work node
$ kubeadm join 192.168.3.167:6443 --token 3zefbt.99e6denc1cxpk9fg \
--discovery-token-ca-cert-hash sha256:8077d4e7dd6eee64a999d56866ae4336073ed5ffc3f23281d757276b08b9b195
Install the yurtctl tool
Install using the following command line yurtctl. The yurtcli CLI tool can install/uninstall OpenYurt and convert standard Kubernetes clusters to OpenYurt clusters.
git clone https://github.com/openyurtio/openyurt.git
cd openyurt
make build WHAT=cmd/yurtctl
Install OpenYurt components
yurtctl convert --deploy-yurttunnel --cloud-nodes oy-master --provider kubeadm\
--yurt-controller-manager-image="openyurt/yurt-controller-manager:v0.5.0"\
--yurt-tunnel-agent-image="openyurt/yurt-tunnel-agent:v0.5.0"\
--yurt-tunnel-server-image="openyurt/yurt-tunnel-server:v0.5.0"\
--node-servant-image="openyurt/node-servant:latest"\
--yurthub-image="openyurt/yurthub:v0.5.0"
Note: need to be
-- yurtctl-servant-image="openyurt/yurtctl-servant:v0.5.0"changed to--node-servant-image="openyurt/node-servant:latest"
Run a simple WebAssembly app
Next, let's run the WebAssembly program as a container in a pod via the Kubernetes cluster. This section will start by pulling a WebAssembly-based container image from Docker hub. If you want to know how to compile, package and publish a WebAssembly program to Docker hub as a container image, see the WasmEdge Book .
Enter the following commands from another terminal window on the master node and start using the cluster.
root@master:~# kubectl cluster-info
Kubernetes master is running at https://192.168.3.167:6443
KubeDNS is running at https://192.168.3.167:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Since kubectl 1.18.9 doesn't have annotations parameter, we need to adjust the command line here. If you are using OpenYurt Experience Center, whose default version is OpenYurt 0.6.0 and Kubernetes 1.20.11, please refer directly to the Kubernetes section in the WasmEdge Book to run wasm applications.
// kubectl 1.18.9
$ sudo kubectl run -it --rm --restart=Never wasi-demo --image=hydai/wasm-wasi-example:with-wasm-annotation --overrides='{"kind":"Pod","metadata":{"annotations":{"module.wasm.image/variant":"compat"}} , "apiVersion":"v1", "spec": {"hostNetwork": true}}' /wasi_example_main.wasm 50000000
// kubectl 1.20.11
$ sudo kubectl run -it --rm --restart=Never wasi-demo --image=hydai/wasm-wasi-example:with-wasm-annotation --annotations="module.wasm.image/variant=compat" --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"hostNetwork": true}}' /wasi_example_main.wasm 50000000
The output of the containerized application is printed to the console. All Kubernetes versions are the same.
Random number: 1123434661
Random bytes: [25, 169, 202, 211, 22, 29, 128, 133, 168, 185, 114, 161, 48, 154, 56, 54, 99, 5, 229, 161, 225, 47, 85, 133, 90, 61, 156, 86, 3, 14, 10, 69, 185, 225, 226, 181, 141, 67, 44, 121, 157, 98, 247, 148, 201, 248, 236, 190, 217, 245, 131, 68, 124, 28, 193, 143, 215, 32, 184, 50, 71, 92, 148, 35, 180, 112, 125, 12, 152, 111, 32, 30, 86, 15, 107, 225, 39, 30, 178, 215, 182, 113, 216, 137, 98, 189, 72, 68, 107, 246, 108, 210, 148, 191, 28, 40, 233, 200, 222, 132, 247, 207, 239, 32, 79, 238, 18, 62, 67, 114, 186, 6, 212, 215, 31, 13, 53, 138, 97, 169, 28, 183, 235, 221, 218, 81, 84, 235]
Printed from wasi: This is from a main function
This is from a main function
The env vars are as follows.
The args are as follows.
/wasi_example_main.wasm
50000000
File content is This is in a file
pod "wasi-demo" deleted
You can now view the pod status via the Kubernetes command line.
crictl ps -a
You can see events in the log from scheduling to running the WebAssembly workload.
CONTAINER IMAGE CREATED STATE
NAME ATTEMPT POD ID
0c176ed65599a 0423b8eb71e31 8 seconds ago Exited wasi-demo
This is done. Next, you can try using OpenYurt to manage your own WasmEdge application .
The future can be expected
Together, WasmEdge and OpenYurt bring the full benefits of Kubernetes to edge computing. The above demo demonstrates a very simple usage scenario, just touching the corner of its huge potential and prospects. Follow both projects on GitHub to stay up to date on our collaboration. Also, if you have an application that benefits from this technology path, we'd love to hear it too!
- WasmEdge GitHub repo: https://github.com/WasmEdge/WasmEdge
- OpenYurt GitHub repo: https://github.com/openyurtio/openyurt