Golang uses CGO&Plugin technology to load C dynamic library at runtime

Introduction to the article

This paper introduces a technology for loading C dynamic library in Golang program at runtime, skips the process of linking C dynamic library in Golang project compilation phase, and improves the flexibility of Golang project development and deployment.

technical background

The Golang program calls the functions of the OpenCC dynamic library to perform the operation of converting traditional Chinese to simplified text. It is necessary to not link the dynamic library at compile time, and only load the OpenCC dynamic library when the program is running.

The OpenCC library is a traditional and simplified conversion program written in C++ and provides a C language API interface. Open source project address: github.com/BYVoid/Open…

CGO technology is a way to let Golang language use C language code, which can link C dynamic library when Golang program is compiled. After decades of development, the C language has a wealth of open source projects. When the Golang ecosystem is not very complete, we often need to use some mature C open source projects.

Plugin is a new plug-in system introduced in Golang 1.8 version, which allows programmers to build loosely coupled modular programs using dynamic link libraries, which are dynamically loaded and bound when the program is running.

This article explains 2 solutions step by step:

Solution 1: Use CGO technology to bind the OpenCC interface at compile time.

Solution 2: Introduce Plugin technology to load dynamic library when the program is running.

Solution 1

Scheme 1 is the original solution, which uses CGO technology and binds the OpenCC interface at compile time.

Golang -> CGO -> libopencc.so

Write the CGO interface definition file:

// opencc.h
#include <stdlib.h>

void* Opencc_New(const char *configFile);
void* Opencc_Delete(void *id);
const char *Opencc_Convert(void *id, const char *input);
void Opencc_Free_String(char *p);
复制代码

// opencc.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "opencc/src/opencc.h"

const char *Convert(const char *input, const char *config) {
        if(strlen(config) > 16) {
                return 0;
        }

        char configFile[256] = "/usr/share/opencc/";
        strcat(configFile, config);
        strcat(configFile, ".json");

        opencc_t p = opencc_open(configFile);
        char *out = opencc_convert_utf8(p, input, strlen(input));
        out[strlen(input)] = '\0';

        opencc_close(p);

        return out;
}

void Convert_free_string(char *p) {
        opencc_convert_utf8_free(p);
}

void* Opencc_New(const char *configFile) {
        return opencc_open(configFile);
}

void Opencc_Delete(void *id) {
        opencc_close(id);
}

const char *Opencc_Convert(void *id, const char *input) {
        char *output = opencc_convert_utf8(id, input, strlen(input));
        output[strlen(input)] = '\0';
        return output;
}

void Opencc_Free_String(char *p) {
        opencc_convert_utf8_free(p);
}
复制代码

// opencc.go
package opencc

import "unsafe"
// #cgo LDFLAGS: -L${SRCDIR}/output/lib -lopencc
// #include "opencc.h"

import "C"

func NewConverter(config string) unsafe.Pointer {
        return C.Opencc_New(C.CString(config))
}

func Convert(p unsafe.Pointer, input string) string {
        inputChars := C.CString(input)
        outputChars := C.Opencc_Convert(p, inputChars)

        defer C.Opencc_Free_String(inputChars)
        defer C.Opencc_Free_String(outputChars)

        result := C.GoString(outputChars)
        return result
}

func Close(p unsafe.Pointer) {
        C.Opencc_Delete(p)
}

func ConvertOneTime(input string, config string) string {
        p := NewConverter(config)
        defer Close(p)
        return Convert(p, input)
}

func ConvertAsync(input string, config string, callback func(output string)) {
        go func() {
                callback(ConvertOneTime(input, config))
        }()
}
复制代码

// test.go 调用OpenCC动态库的函数。
package main

import "fmt"
import "your/repository/go-opencc"

const (
        input = "中国鼠标软件打印机"
        config_s2t = "/usr/share/opencc/s2t.json"
        config_t2s = "/usr/share/opencc/t2s.json"
)

func main() {
        fmt.Println("Test Converter class:")
        c := opencc.NewConverter(config_s2t)
        defer c.Close()
        output := c.Convert(input)
        fmt.Println(output)

        fmt.Println("Test Convert function:")
        s := opencc.Convert(input, config_s2t)
        fmt.Println(s)
        fmt.Println(opencc.Convert(s, config_t2s))

        fmt.Println("Test ConvertAsync function:")
        retChan := make(chan string)
        opencc.ConvertAsync(input, config_s2t, func(output string) {
                retChan <- output
        })

        fmt.Println(<- retChan)
}
复制代码

Option 1, the libopencc.so file can be correctly linked, and the Convert function can be successfully executed to perform traditional and simplified conversion. But there is a problem, the program is not easy to compile on Mac system, and the OpenCC project needs to be installed on the Mac system. If the project calling OpenCC is jointly developed by 10 people, it needs to be compiled on the Mac computer of 10 people, which is difficult for development collaboration and inconvenient for deployment.

Solution 2

The Plugin technology is introduced, and the dynamic library is loaded when the program is running.

Golang -> Plugin -> libgo_opencc.so -> CGO -> libopencc.so

Write Plugin dynamic link library.

// opencc_lib.go
package main

import (
        "unsafe"

        opencc "your/repository/go-opencc"
)

type openccConverter string

// NewConverter 创建Converter
func (s openccConverter) NewConverter(config string) unsafe.Pointer {
        return opencc.NewConverter(config)
}

// Convert 转换函数
func (s openccConverter) Convert(p unsafe.Pointer, input string) string {
        return opencc.Convert(p, input)
}

// Close 释放Converter占用的内存资源(不再使用Converter了)
func (s openccConverter) Close(p unsafe.Pointer) {
        opencc.Close(p)
}

// ConvertOneTime 转换函数(转换一次，该函数每次调用都会加载配置文件，有性能影响)
func (s openccConverter) ConvertOneTime(input string, config string) string {
        return opencc.ConvertOneTime(input, config)
}

// OpenccConverter export symble
var OpenccConverter openccConverter
复制代码

Compile the dynamic library build.sh to create the output directory, and compile and generate the ./output/lib/libgo_opencc.so dynamic library.

#!/bin/bash

mkdir -p output

cd opencc
./build.sh
cd ..

cp -rf opencc/output/* ./output

go build -buildmode=plugin -o ./output/lib/libgo_opencc.so ./lib/opencc_lib.go
复制代码

Use Plugin to load libgo_opencc.so and call OpenCC functions.

package main

import (
        "os"
        "plugin"
        "testing"
        "unsafe"
        "fmt"
)

// 实现 opencc_lib.go 的接口
type OpenccConverter interface {
        NewConverter(config string) unsafe.Pointer
        Convert(p unsafe.Pointer, input string) string
        Close(p unsafe.Pointer)
        ConvertOneTime(input string, config string) string
}

func TestOpenccSo(t *testing.T) {
        openccPlugin, pluginErr := plugin.Open("/your/path/to/go-opencc/output/lib/libgo_opencc.so")
        if pluginErr != nil {
                panic(pluginErr)
        }

        symbol, cerr := openccPlugin.Lookup("OpenccConverter")
        if cerr != nil {
                fmt.Errorf("%+v", cerr)
                os.Exit(1)
        }

        plug, ok := symbol.(OpenccConverter)
        if !ok {
                fmt.Errorf("unexpected type from module symbol")
                os.Exit(1)
        }

        config := "/usr/share/opencc/hk2s.json"
        pointer := plug.NewConverter(config)
        defer plug.Close(pointer)

        input := "傳統漢字"

        output := plug.Convert(pointer, input)

        fmt.Printf("output: %s", output)
}

// 运行测试 TestOpenccSo，输出 “传统汉字”。
复制代码

Option 2 realizes loading the libgo_opencc.so dynamic library through Plugin technology when the program is running, and then linking libgo_opencc.so to libopencc.so to compile Golang programs on Mac and Linux systems without deploying OpenCC on each development machine project. When finally released to the production environment, libgo_opencc.so and libopencc.so will be packaged and released by the compilation and packaging platform.