Click on the blue text
Follow us to make development more interesting
Author: Yan Guojin, Intel Edge Computing Innovation Ambassador
01
Preface
The Intel released OpenVINO™ tool suite is developed based on oneAPI and can accelerate the development of high-performance computer vision and deep learning vision applications. The tool suite is applicable to various Intel platforms from edge to cloud, helping users to develop more accurate images faster. Real world results are deployed into production systems. Through streamlined development workflows, OpenVINO™ empowers developers to deploy high-performance applications and algorithms in the real world.
OpenVINO™ 2023.2 was released on November 16, 2023, and the toolkit brings new capabilities to unlock the full potential of generative artificial intelligence. More generative AI coverage and framework integration to minimize code changes, and expanded model support for direct PyTorch model conversion. Supports more new models, including famous models such as LLaVA, chatGLM, Bark and LCM. Support for a wider range of large language models (LLM) and more model compression technologies, support for runtime inference, support for the following Int4 model compression formats, native Int4 compression through the Neural Network Compression Framework (NNCF), and a series of new features.
Through the latest OpenVINO™ update information, we can see that OpenVINO™ is actually being continuously updated and adapted to more new models. The main reason is that many emerging models may bring more new operators, and OpenVINO™ may not be adapted until subsequent version updates. But what do we do if we need to deploy this deep learning model using OpenVINO™ at this stage?
In fact, for some programming experts and people who have in-depth research on OpenVINO™, they can modify the OpenVINO™ source code, then write their own model operators and add them to OpenVINO™. However, the threshold for developing operators is very high, and for most developers, this is very difficult to achieve.
When deploying the RT-DETR model before, I also encountered the problem that the operator was not supported when the model was deployed using a GPU device in OpenVINO™. Therefore, in this article, I will combine the solution process of this problem to show you how for most developers to solve the problem of operator mismatch during model deployment.
02
Submit Issues
OpenVINO™ is an open source project on GitHub. The source code is open source, so we can download the source code and compile it ourselves, so we can modify the source code. However, for most developers, modifying the source code is difficult to achieve. Open source projects on GitHub provide the Issues function. If developers encounter problems when using it, they can seek help from official developers by submitting Issues.
The process for submitting Issues is as follows:
01
Visit the OpenVINO™ Factory Issues page and create a [new Issues]
OpenVINO™ Factory Library Issues page:
https://github.com/openvinotoolkit/openvino/issues
02
Select an Issues type, you can choose according to your own issue type
03
After selecting the type, you can fill in the relevant content according to your own problem situation.
04
Taking this issue as an example, let’s briefly introduce the content of Issues:
First, fill in the basic content, which mainly includes the OpenVINO™ version, development platform, inference equipment, model framework, deployed model and a brief description of the problem. The submission for this issue looks like this:
The next step is to retell the process of the problem in detail. In order to allow developers to understand the problem more clearly, this does not need to be written in great detail to ensure that developers can completely reproduce the problem. If the reproduction steps are relatively simple, you can describe them in detail here. If the problem is more complex and there are multiple files, you need to create a GitHub repository to store related files.
The last step is to add error log output, mainly to allow developers to confirm whether the problem has been successfully reproduced based on the log, or to locate the problem based on the log output.
After submitting Issues, OpenVINO™ official staff will assign developers to solve the problem based on the content of the issue. At this time, we only need to wait for the official response from OpenVINO™.
03
Compile OpenVINO™ source code
In the previous step, we asked relevant parties for help, and we will get the official answer soon. However, if the official answer involves source code modification, most developers may not be able to start. For changes involving source code, if you want to If you use it directly in the project, you need to compile the source code yourself. Because if the official solution to the problem you raised is valuable, it may be fixed in subsequent versions and released to the latest version. But for us users, we need to use it immediately and cannot wait until the next version is released. Therefore, we will combine the official replies and changes and start from the source code step by step based on the Windows platform to compile and use the source code.
The official source code compilation process link is:
https://github.com/openvinotoolkit/openvino/blob/master/docs/dev/build_windows.md
3.1 Download source code
OpenVINO™ source code can be downloaded through Git. Use Git to run the following code in sequence:
git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --initSwipe left to see more
Due to network problems, the download here may be slow, but you must ensure that some additional dependencies downloaded above can be downloaded completely, otherwise errors will occur during subsequent compilation. If there are missing items when generating compiled files later, the main reason is that some files were not downloaded due to network reasons. I also downloaded them several times before downloading the complete files.
3.2 Modify source code
In the previous step, we have downloaded the source code. The next step is to modify the source code according to the official modification opinions. As shown in the figure below, this screenshot shows the official source code modification opinions:
In this return opinion, we need to add some code to the source code, so here you can modify the source code according to the situation.
3.3 Source code compilation
The source code has been downloaded and modified in the previous step, and you can compile it next. Here we use Cmake+vs2022 to compile, compile and generate the VS2022 project through Cmake, and then use VS to compile the project source code.
First run the following two commands in sequence:
mkdir build && cd build
cmake -G "Visual Studio 17 2022" <path/to/openvino> -DCMAKE_BUILD_TYPE=DebugSwipe left to see more
Here you need to replace "<path/to/openvino>" with your own OpenVINO™ path. After running the first command, the build folder will be created and switched to this folder. After running the second command, as shown in the figure below, it will be compiled according to the CMakeLists file in the project, and the final successful compilation output is as shown in the second figure.
If you cannot get the effect shown above, it means there is a problem in compilation, and you need to go back and look for related errors. Following Jun's previous compilation experience, the main problem occurred in the first step. The main reason was that the relevant configuration was not downloaded due to network problems. After the final compilation is successful, the OpenVINO.sln project solution will appear in the build file.
Next, we use VS2022 to open the solution, and then select the version of the generated dynamic link library by setting Debug or Release, then right-click the ALL_BUILD project, click Generate, and then run the project for source code compilation.
Friendly reminder, this step of source code compilation takes a long time to generate due to the code file angle involved.
3.4 Get the compiled dynamic link library
After the project runs successfully, we can find the compiled dynamic link library file in the following path. It can be seen that the .dll file and .lib file generated by the compilation are in the official OpenVINO™ distribution runtime we downloaded. The files are consistent, so in subsequent use, we only need to replace the dynamic link library file generated by our compilation into the current project.
04
Actual project testing
This case is mainly based on the RT-DETR model using GPU deployment as an example. Finally, we returned to this case to test whether our problem was solved. Regarding the process of deploying the RTDETR model in C++, please refer to my previous article: " Deploying RT-DETR model based on OpenVINO™ C++ API | Developer practice》, so I won’t explain too much about C++ project configuration here.
As shown in the picture below, this picture is the result of our use of the current release version OpenVINO™ 2023.2. Here we use the GPU.0 integrated graphics card to run inference. It can be seen that the model will have errors during the inference stage, causing the program to crash.
Next, we use the dynamic link library we compiled and generated above to run the program. It can be seen that when using the GPU.0 integrated graphics card to run inference, the error has disappeared, and the model inference is successfully run, and the results are printed.
Through comparative testing, it shows that the official solution is correct. We have also successfully solved the current problem based on the official solution. We can use it in advance without waiting for the official release of the next version, which is convenient for everyone.
05
Summarize
In this article, we combine a model deployment case to explain to you how to solve problems encountered by us users with the help of official developers when problems such as operator mismatch occur when using OpenVINO™. question. In addition, through this method, we also reported bugs in the code to OpenVINO™. While solving our problems, we also actively made our own contributions to open source.
OpenVINO™
--END--
你也许想了解(点击蓝字查看)⬇️➡️ OpenVINO™ 2023.2 发布：让生成式 AI 在实际场景中更易用➡️ 开发者实战 | 介绍OpenVINO™ 2023.1：在边缘端赋能生成式AI➡️ 基于 ChatGLM2 和 OpenVINO™ 打造中文聊天助手➡️ 基于 Llama2 和 OpenVINO™ 打造聊天机器人➡️ OpenVINO™ DevCon 2023重磅回归！英特尔以创新产品激发开发者无限潜能➡️ 5周年更新 | OpenVINO™ 2023.0，让AI部署和加速更容易➡️ OpenVINO™5周年重头戏！2023.0版本持续升级AI部署和加速性能➡️ OpenVINO™2023.0实战 | 在 LabVIEW 中部署 YOLOv8 目标检测模型➡️ 开发者实战系列资源包来啦！➡️ 以AI作画，祝她节日快乐；简单三步，OpenVINO™ 助你轻松体验AIGC
➡️ 还不知道如何用OpenVINO™作画？点击了解教程。扫描下方二维码立即体验
OpenVINO™ 工具套件 2023.2Click to read the original text and experience OpenVINO 2023.2 now
The article is so exciting, are you “reading” it?