How to extend WebAssembly on the server side through Host Function

Host Functions are the main way to extend WebAssembly today. This article will take you to open a new world of WebAssembly through two examples of Host Function!

Author: DarumaDocker, mainly responsible for the development of WasmEdge-bindgen .

WebAssembly was originally developed from the browser. When Wasm slowly migrated from the browser to the server, a major problem it faced was incomplete functions and limited capabilities. The proposal of WASI is expected to solve these problems, but the development and implementation of standards are usually slow.

What if you're in a hurry to use a feature? The answer is to use the Host Function to customize your WebAssembly Runtime.

What is Host Function

As the name suggests, Host Function is a function defined in the Host program. For Wasm, Host Function can importbe module, and then can be called when Wasm is running.

The current capabilities of Wasm are limited, but those things that Wasm cannot do by itself can be solved by the Host Function, which greatly expands the capabilities of Wasm.

The extensions made by WasmEdge beyond the standard are basically done by relying on the Host Function. For example, the Tensorflow API provided by WasmEdge is implemented using the Host Function, thus achieving the goal of running AI inference at native speed.

Networking sockets are also implemented using host functions, so we can run asynchronous HTTP clients and servers in WasmEdge , making up for WebAssembly's shortcomings on the network.

Another example is that Fastly uses Host Function to add interfaces such as Http Request and Key-value store to Wasm, thereby adding extended functions.

How to Write a Simple Host Function

Let's start with the simplest example and see how to write a Host function in a Go program.

Let's start by writing a simple rust program. International practice, Cargo.tomlno less.

Cargo.toml

[package]
name = "rust_host_func"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib", "rlib"]

[dependencies]

Let's take a look at what Rust code looks like.

lib.rs

extern "C" {
	fn add(a: i32, b: i32) -> i32;
}

#[no_mangle]
pub unsafe extern fn run() -> i32 {
	add(1, 2)
}

The addfunction is declared in extern "C", which is a Host Function. We compile this Rust program to wasm using the following command:

cargo build --target wasm32-wasi --release

Then we use wasm2watto view the import section of the wasm file:

wasm2wat target/wasm32-wasi/release/rust_host_func.wasm | grep import

The output is as follows:

  (import "env" "add" (func $add (type 0)))

You can see that the addfunction is placed in envthe import section of the module whose default name is .

Next, let's see how to use the WasmEdge-go SDK to execute this wasm program.

hostfunc.go

package main

import (
	"fmt"
	"os"

	"github.com/second-state/WasmEdge-go/wasmedge"
)

func add(_ interface{}, _ *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// 将从 wasm 传过来的两个参数做加法运算
	return []interface{}{params[0].(int32) + params[1].(int32)}, wasmedge.Result_Success
}

func main() {
	vm := wasmedge.NewVM()
	
	// 使用默认名称 env 构建导入段对象
	obj := wasmedge.NewImportObject("env")

	// 构建 Host Function 的参数和返回值类型
	funcAddType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		})
	hostAdd := wasmedge.NewFunction(funcAddType, add, nil, 0)
	
	// 将 Host Function 加入到导入段对象中
	// 注意第一个参数 `add` 是 rust 中定义的外部函数的名称
	obj.AddFunction("add", hostAdd)

	// 注册导入段对象
	vm.RegisterImport(obj)

	// 加载, 验证并实例化 wasm 程序
	vm.LoadWasmFile(os.Args[1])
	vm.Validate()
	vm.Instantiate()

	// 执行 wasm 导出的函数并取得返回值
	r, _ := vm.Execute("run")
	fmt.Printf("%d", r[0].(int32))

	obj.Release()
	vm.Release()
}

Compile and execute:

go build
./hostfunc rust_host_func.wasm

program output 3.

In this way, we have completed the simplest example of defining Function in Host and calling it in wasm.

Let's try to do something more interesting with the Host Function.

pass complex types

Restricted by the data type in Wasm, Host Function can only pass a few basic types of data such as int32, which will greatly limit the application scope of Host Function. Is there any way for us to pass complex data types such as string data? The answer is of course, let's see how to do it through an example.

In this example, we want to count https://www.google.comthe number of googleoccurrences of in the source code of the web page. The source code of the example is here .

Let's go to Rust code first. Cargo.tomlis essential, I just omitted it here.

lib.rs

extern "C" {
	fn fetch(url_pointer: *const u8, url_length: i32) -> i32;
	fn write_mem(pointer: *const u8);
}

#[no_mangle]
pub unsafe extern fn run() -> i32 {
	let url = "https://www.google.com";
	let pointer = url.as_bytes().as_ptr();

	// call host function to fetch the source code, return the result length
	let res_len = fetch(pointer, url.len() as i32) as usize;

	// malloc memory
	let mut buffer = Vec::with_capacity(res_len);
	let pointer = buffer.as_mut_ptr();

	// call host function to write source code to the memory
	write_mem(pointer);

	// find occurrences from source code
	buffer.set_len(res_len);
	let str = std::str::from_utf8(&buffer).unwrap();
	str.matches("google").count() as i32
}

In this code, two Host Functions are introduced:

• fetchUsed to send http requests to get web page source code
• write_memMemory used to write web page source code to wasm

You may have seen that to pass a string in Host Function, it is actually achieved by passing the memory pointer and length where the string is located. fetchReceive two parameters, they are https://www.google.comthe pointer and the byte length respectively.

fetchAfter obtaining the source code, return the byte length of the source code as the return value. After allocating memory of this length, Rust passes the memory pointer to write_memthe host and writes the source code to this memory, thereby achieving the purpose of returning a string.

The compilation process is the same as above and will not be repeated. Next, we will show how to use the WasmEdge-go SDK to execute this Wasm program.

hostfun.go

package main

import (
	"fmt"
	"io"
	"os"
	"net/http"

	"github.com/second-state/WasmEdge-go/wasmedge"
)

type host struct {
	fetchResult []byte
}

// do the http fetch
func fetch(url string) []byte {
	resp, err := http.Get(string(url))
	if err != nil {
		return nil
	}
	defer resp.Body.Close()
	body, err := io.ReadAll(resp.Body)
	if err != nil {
		return nil
	}

	return body
}

// Host function for fetching
func (h *host) fetch(_ interface{}, mem *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// get url from memory
	pointer := params[0].(int32)
	size := params[1].(int32)
	data, _ := mem.GetData(uint(pointer), uint(size))
	url := make([]byte, size)

	copy(url, data)

	respBody := fetch(string(url))

	if respBody == nil {
		return nil, wasmedge.Result_Fail
	}

	// store the source code
	h.fetchResult = respBody

	return []interface{}{len(respBody)}, wasmedge.Result_Success
}

// Host function for writting memory
func (h *host) writeMem(_ interface{}, mem *wasmedge.Memory, params []interface{}) ([]interface{}, wasmedge.Result) {
	// write source code to memory
	pointer := params[0].(int32)
	mem.SetData(h.fetchResult, uint(pointer), uint(len(h.fetchResult)))

	return nil, wasmedge.Result_Success
}

func main() {
	conf := wasmedge.NewConfigure(wasmedge.WASI)
	vm := wasmedge.NewVMWithConfig(conf)
	obj := wasmedge.NewImportObject("env")

	h := host{}
	// Add host functions into the import object
	funcFetchType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		})

	hostFetch := wasmedge.NewFunction(funcFetchType, h.fetch, nil, 0)
	obj.AddFunction("fetch", hostFetch)

	funcWriteType := wasmedge.NewFunctionType(
		[]wasmedge.ValType{
			wasmedge.ValType_I32,
		},
		[]wasmedge.ValType{})
	hostWrite := wasmedge.NewFunction(funcWriteType, h.writeMem, nil, 0)
	obj.AddFunction("write_mem", hostWrite)

	vm.RegisterImport(obj)

	vm.LoadWasmFile(os.Args[1])
	vm.Validate()
	vm.Instantiate()

	r, _ := vm.Execute("run")
	fmt.Printf("There are %d 'google' in source code of google.com\n", r[0])

	obj.Release()
	vm.Release()
	conf.Release()
}

With an understanding of Rust code, this go code is actually very easy to understand. The key is to access Wasm memory:

• mem.GetData(uint(pointer), uint(size))Get the url of a web page in Wasm
• mem.SetData(h.fetchResult, uint(pointer), uint(len(h.fetchResult)))Write webpage source code into wasm memory

The compilation and execution steps of this example are exactly the same as the previous example, and the final execution result is:

There are 79 'google' in source code of google.com

Epilogue

Through the above two examples, I believe you have a preliminary impression of Host Function. Although the development experience is not ideal due to the many limitations of Wasm, as we continue to improve the tools and libraries, it will bring infinite possibilities to the application scenarios of Wasm.

Welcome to continue to pay attention to the WasmEdge project. If you think WasmEdge is good, you are also welcome to star, thank you.

About WasmEdge

WasmEdge is a lightweight, secure, high-performance, real-time software container and runtime environment. Currently a CNCF sandbox project. WasmEdge is used in SaaS, cloud native, service mesh, edge computing, automotive and other fields.

• GitHub：https://github.com/WasmEdge/WasmEdge
• Official website: https://wasmedge.org/
• Discord group: https://discord.gg/JHxMj9EQbA
• Documentation: https://wasmedge.org/book/en

{{o.name}}

{{m.name}}