Author of the article: Yan Guojin, Intel Edge Computing Innovation Ambassador
Technical guidance: Wu Zhuo, Li Yiwei
The OpenVINO™ toolkit can speed up the development of deep learning vision applications and help users deploy AI models to production systems more conveniently and quickly on various Intel platforms from edge to cloud.
C# is a safe, stable, simple, and elegant object-oriented programming language derived from C and C++. It combines the simple visual operation of VB and the high operating efficiency of C++, and becomes the first choice for .NET development language. As an artificial intelligence developer, if you want to use OpenVINO™ on the C# side, OpenVINOSharp will be your first choice , and a NuGet package has been made to realize the one-stop installation and use of OpenVINO ™ on the C# side .
OpenVINOSharp GitHub: https://github.com/guojin-yan/OpenVinoSharp
OpenVINOSharp refers to the OpenVINO™ C++ API when making it, so it is very friendly to people who have used OpenVINO™ before. The following table shows us the corresponding relationship between C# and C++ API
| Class |
C++ API |
C# API |
illustrate |
| Core class |
ov::Core |
Core |
OpenVINO runtime core entity classes |
| Model class |
ov::Model |
Model |
User-defined model class |
| CompiledModel class |
ov::CompiledModel |
CompiledModel |
Compiled model class |
| Output class |
ov:: Output<ov::Node> |
Output |
Handle class for node output |
| Input class |
ov:: Input <ov::Node> |
Input |
Handle class for node input |
| InferRequest class |
ov::InferRequest |
ov::InferRequest |
Class to run inference requests asynchronously or synchronously |
| Tensor class |
ov::Tensor |
Tensor |
Tensor |
| Shape class |
ov::Shape |
Shape |
tensor shape class |
In this article, based on the general steps of model deployment, the use of method functions from model loading to inference will be demonstrated and compared with the C++ API.
1.1 Install OpenVINOSharp
OpenVINOSharp supports the NuGet package installation method, which is simpler than the installation process in C++, and the package contains the OpenVINO™ 2023.0 release version , which can be used directly after installation through NuGet.
If you use Visual Studio to compile the project, you can install it directly through the NuGet package management function:
If you install it through the dotnet command, you can install it through the following statement:
dotnet add package OpenVinoSharp.win1.2 Import assembly
All OpenVINOSharp assemblies are under the namespace OpenVinoSharp, so if you want to use OpenVINOSharp, you need to introduce the namespace first:
using OpenVinoSharp;1.3 Initialize the OpenVINO runtime kernel
The Core class represents an OpenVINO runtime core entity. Subsequent methods such as reading models and loading models need to be created through the Core class. When encapsulating the C# API, in order to correspond to the C++ API, the Core class is also encapsulated and encapsulated The corresponding method in the C++ API
Initialization method in C#:
Core core = new Core();Initialization method in C++:
ov::Core core;1.4 Load and get model information
1.4.1 Load model
After the OpenVINO™ 2022.1 version is updated and loaded, the following are the API methods used:
| API |
effect |
| Core.read_model () |
Load the model from the hard disk into memory and return the Model object. |
How to load a model in C#:
Model model = core.read_model(model_path);Initialization method in C++:
std::shared_ptr<ov::Model> model = core.read_model(model_path);1.4.2 Get model information
Both the Model object obtained through the Core.read_model () method and the CompiledModel object obtained through the Core.compile_model () method support direct access to attributes to obtain input and output layer information. Take the Model object to obtain model information as an example, the following is the API method used:
| API |
effect |
| Model.get_friendly_name() |
Get the friendly name of the model. |
| Model.input() |
Get the input layer of the model and return the Input object. |
| Model.output() |
Get the output layer of the model and return the Output object. |
Input/Output mainly encapsulates the model network layer, and the detailed information of the model can be obtained through the following API:
| API |
effect |
| Output.get_any_name() |
Get the name of the model network layer. |
| Output.get_element_type() |
Obtain the data type of the network layer of the model and return the OvType object. OvType mainly encapsulates the basic data types of the network. |
| Output.get_shape() |
Get the shape of the network layer of the model and return the Shape object, which encapsulates the shape array of the network layer. |
In C#, through the following code, you can directly get the input of the model, the input layer and the friendly name of the model:
string model_name = model.get_friendly_name();
Input input = model.input();
Output output = model.output();Then print the model specific information to the console page:
Console.WriteLine("Model name: {0}", model_name);
Console.WriteLine("/------- [In] -------/");
Console.WriteLine("Input name: {0}", input.get_any_name());
Console.WriteLine("Input type: {0}", input.get_element_type().to_string());
Console.WriteLine("Input shape: {0}", input.get_shape().to_string());
Console.WriteLine("/------- [Out] -------/");
Console.WriteLine("Output name: {0}", output.get_any_name());
Console.WriteLine("Output type: {0}", output.get_element_type().to_string());
Console.WriteLine("Output shape: {0}", output.get_shape().to_string());Obtain model network layer information as follows:
Model name: torch_jit
/------- [In] -------/
Input name: data
Input type: float
Input shape: [1,3,224,224]
/------- [Out] -------/
Output name: prob
Output type: float
Output shape: [1,1000]The same output information, we use the C++ API to achieve the following:
std::cout << "Model name: " << model->get_friendly_name() << std::endl;
ov::Output<ov::Node> input = model->input();
std::cout << "/------- [In] -------/" << std::endl;
std::cout << "Input name: " << input.get_any_name() << std::endl;
std::cout << "Input type: " << input.get_element_type().c_type_string() << std::endl;
std::cout << "Input shape: " << input.get_shape().to_string() << std::endl;
ov::Output<ov::Node> output = model->output();
std::cout << "/------- [Out] -------/" << std::endl;
std::cout << "Output name: " << output.get_any_name() << std::endl;
std::cout << "Output type: " << output.get_element_type().c_type_string() << std::endl;
std::cout << "Output shape: " << output.get_shape().to_string() << std::endl;1.5 Compile the model and create an inference request
After reading the local model, call the model compilation method to compile the model into a compile_model object that can be executed on the target device, and create an inference request object for inferring the compiled model through this object. The following are the API methods used:
| API |
effect |
| Core.compile_model() |
Compiles the model into a compile_model object that can be executed on the target device. |
| CompiledModel.create_infer_request() |
Creates an inference request object used to infer the compiled model. The created request has input and output tensors allocated. |
How to compile the model and create inference requests in C#:
CompiledModel compiled_model = core.compile_model(model, "AUTO");
InferRequest infer_request = compiled_model.create_infer_request();Use the C++ API to compile the model and create an inference request:
CompiledModel compiled_model = core.compile_model(model, "AUTO");
InferRequest infer_request = compiled_model.create_infer_request();1.6 TensorTensor _
1.6.1 Obtaining and setting tensors
After creating an inference request, the system will automatically create and allocate input and output tensors. The tensors can be obtained through the InferRequest object, and the tensors can be customized and loaded to the specified node of the model; the sequence number and name can be output according to the input of the tensor. As well as obtaining and setting model node Node objects, the main C# API is as follows:
| API |
effect |
| InferRequest.set_tensor() |
Set the input/output tensors to infer. |
| InferRequest.set_input_tensor() |
Set the input tensor to infer. |
| InferRequest.set_output_tensor() |
Set the output tensor to infer |
| InferRequest.get_tensor() |
Get input/output tensors for inference. |
| InferRequest.get_input_tensor() |
Get an input tensor for inference. |
| InferRequest.get_output_tensor() |
Get the output tensor for inference. |
1.6.2 Acquisition and setting of tensor information
The main information contained in a tensor includes the shape of the tensor (Shape), the data format of the tensor (OvType-> element.Type), and the memory data in the tensor. The parameters of a tensor can be manipulated through the following API methods:
| API |
effect |
| Tensor.set_shape () |
Give the tensor a new shape. |
| Tensor.get_shape() |
Get the shape of the tensor. |
| Tensor.get_element_type() |
Get the data type of the tensor. |
| Tensor.get_size() |
Get the data length of the tensor. |
| Tensor.get_byte_size() |
Get the byte size of the tensor. |
| Tensor.data() |
Get the memory address of the tensor. |
| Tensor.set_data<T>() |
Load data of the specified type into tensor memory. |
| Tensor.get_data<T>() |
Read data of the specified type from the tensor. |
The above methods are some basic operations on tensors. Except for set_data and get_data, which are unique to OpenVINOSharp, other interfaces are consistent with the C++ API.
1.7 Load inference data
1.7.1 获取输入张量
对于单输入的模型可以直接通过get_input_tensor()方法获得，并调用Tensor的相关方法获取Tensor的相关信息，C# 代码如下所示：
Tensor input_tensor = infer_request.get_input_tensor();
Console.WriteLine("/------- [Input tensor] -------/");
Console.WriteLine("Input tensor type: {0}", input_tensor.get_element_type().to_string());
Console.WriteLine("Input tensor shape: {0}", input_tensor.get_shape().to_string());
Console.WriteLine("Input tensor size: {0}", input_tensor.get_size());获取输出结果为：
/------- [Input tensor] -------/
Input tensor type: f32
Input tensor shape: Shape : {1, 3, 224, 224}
Input tensor size: 150528对于上述的同样输出内容，我们也可以通过C++ API 实现，C++ 代码如下：
ov::Tensor input_tensor = infer_request.get_input_tensor();
std::cout << "/------- [Input tensor] -------/" << std::endl;
std::cout << "Input tensor type: " << input_tensor.get_element_type().c_type_string() << std::endl;
std::cout << "Input tensor shape: " << input_tensor.get_shape().to_string() << std::endl;
std::cout << "Input tensor size: " << input_tensor.get_size() << std::endl;
1.7.2 添加推理数据
这一步主要是将处理好的图片数据加载到Tensor数据内存中，OpenVINO的API中提供了访问内存地址的接口，可以获取数据内存首地址，不过为了更好的加载推理数据，我们此处封装了set_data<T>()方法，可以实现将处理后的图片数据加载到数据内存上。在C#中的代码为：
Mat input_mat = new Mat();
Shape input_shape = input_tensor.get_shape();
long channels = input_shape[1];
long height = input_shape[2];
long width = input_shape[3];
float[] input_data = new float[channels * height * width];
Marshal.Copy(input_mat.Ptr(0), input_data, 0, input_data.Length);
input_tensor.set_data(input_data);下面是在C++中实现上述功能的代码：
cv::Mat input_mat；
float* input_data = input_tensor.data<float>();
ov::Shape input_shape = input_tensor.get_shape();
size_t channels = input_shape[1];
size_t height = input_shape[2];
size_t width = input_shape[3];
for (size_t c = 0; c < channels; ++c) {
for (size_t h = 0; h < height; ++h) {
for (size_t w = 0; w < width; ++w) {
input_data[c * height * width + h * width + w] = input_mat.at<cv::Vec<float, 3>>(h, w)[c];
}
}
}1.8 模型推理
在加载完推理数据后，就可以调用模型推理的API方法推理当前数据，主要使用到的API方法为：
| API |
作用 |
| InferRequest.infer() |
在同步模式下推断指定的输入。 |
调用该方法也较为简单，只需要调用该API接口即可，在C#中的代码为：
infer_request.infer();C++中的代码与C++中一致。
1.9 获取推理结果
对于单输出的模型可以直接通过get_output_tensor()方法获得，并调用Tensor的相关方法获取Tensor的相关信息，C# 代码如下所示：
Tensor output_tensor = infer_request.get_output_tensor();
Console.WriteLine("/------- [Output tensor] -------/");
Console.WriteLine("Output tensor type: {0}", output_tensor.get_element_type().to_string());
Console.WriteLine("Output tensor shape: {0}", output_tensor.get_shape().to_string());
Console.WriteLine("Output tensor size: {0}", output_tensor.get_size());获取输出output_tensor信息为：
/------- [Output tensor] -------/
Output tensor type: f32
Output tensor shape: Shape : {1, 1000}
Output tensor size: 1000对于输出Tensor，我们只需要读取输出内存上的数据即可，此处我们封装了get_data<T>()方法，可以直接获取输出内存上的数据，在C#中的代码为：
float[] result = output_tensor.get_data<float>(1000);同样获取推理结果，在C++中的代码为：
const float* output_data = output_tensor.data<const float>();
float result[1000];
for (int i = 0; i < 1000; ++i) {
result[i] = *output_data;
output_data++;
}在获取结果后，后续的处理需要根据模型的输出类型做相应的处理。
1.10 释放分配的内存
由于C#在封装时采用的C API 接口实现的，因此在C#中会产生较多的 非托管内存，若该对象出现循环重复创建，会导致过多的内存未释放导致内存泄漏，因此对于临时创建的对象在使用后要即使销毁，销毁方式也较为简单，只需要调用对象的dispose()方法即可。
output_tensor.dispose();
input_shape.dispose();
infer_request.dispose();
compiled_model.dispose();
input.dispose();
output.dispose();
model.dispose();
core.dispose();1.11 Yolov8 classification model example
The following code shows the complete code of the Yolov8 classification model using the OpenVINOSharp API method to deploy the model:
using OpenCvSharp;
using OpenCvSharp.Dnn;
using OpenVinoSharp;
using System.Data;
using System.Runtime.InteropServices;
namespace test_openvinosharp_api
{
internal class Program
{
static void Main(string[] args)
{
string model_path = "E:\\GitSpace\\OpenVinoSharp\\model\\yolov8\\yolov8s-cls.xml";
Core core = new Core(); // 初始化推理核心
Model model = core.read_model(model_path); // 读取本地模型
CompiledModel compiled_model = core.compile_model(model, "AUTO"); // 便哟模型到指定设备
// 获取模型的输入输出信息
Console.WriteLine("Model name: {0}", model.get_friendly_name());
Input input = compiled_model.input();
Console.WriteLine("/------- [In] -------/");
Console.WriteLine("Input name: {0}", input.get_any_name());
Console.WriteLine("Input type: {0}", input.get_element_type().to_string());
Console.WriteLine("Input shape: {0}", input.get_shape().to_string());
Output output = compiled_model.output();
Console.WriteLine("/------- [Out] -------/");
Console.WriteLine("Output name: {0}", output.get_any_name());
Console.WriteLine("Output type: {0}", output.get_element_type().to_string());
Console.WriteLine("Output shape: {0}", output.get_shape().to_string());
// 创建推理请求
InferRequest infer_request = compiled_model.create_infer_request();
// 获取输入张量
Tensor input_tensor = infer_request.get_input_tensor();
Console.WriteLine("/------- [Input tensor] -------/");
Console.WriteLine("Input tensor type: {0}", input_tensor.get_element_type().to_string());
Console.WriteLine("Input tensor shape: {0}", input_tensor.get_shape().to_string());
Console.WriteLine("Input tensor size: {0}", input_tensor.get_size());
// 读取并处理输入数据
Mat image = Cv2.ImRead(@"E:\GitSpace\OpenVinoSharp\dataset\image\demo_7.jpg");
Mat input_mat = new Mat();
input_mat = CvDnn.BlobFromImage(image, 1.0 / 255.0, new Size(224, 224), 0, true, false);
// 加载推理数据
Shape input_shape = input_tensor.get_shape();
long channels = input_shape[1];
long height = input_shape[2];
long width = input_shape[3];
float[] input_data = new float[channels * height * width];
Marshal.Copy(input_mat.Ptr(0), input_data, 0, input_data.Length);
input_tensor.set_data(input_data);
// 模型推理
infer_request.infer();
// 获取输出张量
Tensor output_tensor = infer_request.get_output_tensor();
Console.WriteLine("/------- [Output tensor] -------/");
Console.WriteLine("Output tensor type: {0}", output_tensor.get_element_type().to_string());
Console.WriteLine("Output tensor shape: {0}", output_tensor.get_shape().to_string());
Console.WriteLine("Output tensor size: {0}", output_tensor.get_size());
// 获取输出数据
float[] result = output_tensor.get_data<float>(1000);
List<float[]> new_list = new List<float[]> { };
for (int i = 0; i < result.Length; i++)
{
new_list.Add(new float[] { (float)result[i], i });
}
new_list.Sort((a, b) => b[0].CompareTo(a[0]));
KeyValuePair<int, float>[] cls = new KeyValuePair<int, float>[10];
for (int i = 0; i < 10; ++i)
{
cls[i] = new KeyValuePair<int, float>((int)new_list[i][1], new_list[i][0]);
}
Console.WriteLine("\n Classification Top 10 result : \n");
Console.WriteLine("classid probability");
Console.WriteLine("------- -----------");
for (int i = 0; i < 10; ++i)
{
Console.WriteLine("{0} {1}", cls[i].Key.ToString("0"), cls[i].Value.ToString("0.000000"));
}
// 销毁非托管内存
output_tensor.dispose();
input_shape.dispose();
infer_request.dispose();
compiled_model.dispose();
input.dispose();
output.dispose();
model.dispose();
core.dispose();
}
}
}1.12 Summary
In this article, we demonstrate the use of OpenVINOSharp API based on the model inference process, and compare it with OpenVINO C++ API to show the difference between the use of OpenVINOSharp API and C++ API. This also has a positive impact on the development of C++ API. The developers are very friendly and it will be very easy to get started.
In this article, we only show the basic model inference process code, and also test each API. For other more advanced API methods, we will continue to test other API methods in the future to show developers their usage.
In general, OpenVINOSharp currently fully supports installation and use in the Windows environment . Developers are welcome to install and use it. If you have relevant questions or optimization methods, you are also welcome to provide comments and guidance.