1. Application scenarios of portrait matting
1. Image editing: In image editing software, portrait matting is one of the commonly used operations. Through the cutout, the characters can be separated from the original background and placed in the new background, so as to achieve the effect of changing the background of the characters. This has a wide range of applications in the fields of film post-production special effects, game role replacement, etc.
2. Video special effects: In video editing and special effects production, portrait matting is also one of the key technologies. By cutting out and tracking the characters in the video frame in real time, special effects such as disappearing characters and changing clothes can be realized, greatly enriching the visual effect of the video. 3. AR special effects: In the production of special effects in AR (Augmented Reality) and still images, picking people or target objects is a very important step. Only by accurately picking out people or objects can they be replaced or tracked in real time in real-time scenes, and various interesting AR special effects can be realized. This is reflected in many mobile phone AR cosmetics and interfaces. 4. Face recognition: In the face recognition system, detecting and locating faces is the first step. Portrait matting technology is frequently used to assist the detection and locating process to improve the performance and accuracy of the system. 5. Virtual makeup: Through portrait matting, the face area of the character can be separated. Then add various makeup and decorations on the face area, and then fuse them back to the original image to achieve the effect of wearing various dynamic makeup and decorations on the characters, which is widely demanded in the field of modern makeup. In summary, portrait matting has very important application value in the fields of image processing, computer vision and artificial intelligence. With the advancement of related technologies, the effect of portrait matting has been continuously improved, and the scope of application has also continued to expand. As a basic technology, it will serve more diversified application scenarios. This also brings many opportunities and challenges to researchers and developers engaged in related industries.
2. Cutout effect
Traverse folder cutout
Video portrait cutout
3. Deployment
Download the source code from reference URL 1, download onnxruntime-win-x64-gpu-1.9.0 according to the Readme file, and add (opencv4.5.5 and onnxruntime-win-x64-gpu-1.9.0) include directory, library directory, windows Runtime library directory, linker - input - additional dependencies (opencv_world455.lib, onnxruntime.lib).
Configuration property -Vcpkg-Use Vcpkg is set to No
Modify the cuda reference in the header file
//#include <cuda_provider_factory.h>
#include <tensorrt_provider_factory.h>Modify the Detect_video.cpp file
#include "Detect_video.h"
#include "Detect.h"
// 调用摄像头或者读取本地视频文件，进行抠图效果展示
void detect_video(float downsample_ratio, Ort::Session& session, string choose,
std::vector<std::vector<int64_t>>& dynamic_input_node_dims,
std::vector<float>& dynamic_src_value_handler,
std::vector<float>& dynamic_r1i_value_handler,
std::vector<float>& dynamic_r2i_value_handler,
std::vector<float>& dynamic_r3i_value_handler,
std::vector<float>& dynamic_r4i_value_handler,
std::vector<float>& dynamic_dsr_value_handler,
std::vector<const char*>& input_node_names,
std::vector<const char*>& output_node_names,
bool& context_is_update,
unsigned int& num_inputs,
unsigned int& num_outputs
)
{
double inferenceTime = 0.0;
cv::VideoCapture video_capture;
cv:String outpath;
// 选择视频地址抠图或者摄像头抠图
if (choose == "movie") {
cout << "视频地址：" << endl;
cin >> choose;
video_capture.open(choose);
outpath = choose + "_out.mp4";
}
else
{
cout << "请输入调用摄像头的序号：" << endl;
int num = 0;
cin >> num;
video_capture.open(num);
outpath= cv::format("%d", num) + "_out.mp4";
}
if (!video_capture.isOpened())
{
std::cout << "Can not open video" << "\n";
return;
}
// 获取视频信息
int fps = video_capture.get(cv::CAP_PROP_FPS);
cv::Size size = cv::Size((int)video_capture.get(cv::CAP_PROP_FRAME_WIDTH),
(int)video_capture.get(cv::CAP_PROP_FRAME_HEIGHT));
// 创建视频写入对象
cv::VideoWriter outputVideo;
//cv:String outpath = choose == "movie" ? choose + "_out.mp4" : cv::format("%d",num) + "_out.mp4";
outputVideo.open(outpath,
cv::VideoWriter::fourcc('M', 'J', 'P', 'G'),
fps, size, true);
if (!outputVideo.isOpened()) {
printf("Failed to open video writer!");
return ;
}
// 2. matting loop
cv::Mat mat,dst1,dst2;
while (video_capture.read(mat))
{
//开始时间
double t1 = static_cast<double>(cv::getTickCount());
cv::resize(mat, dst1, cv::Size(), 0.5, 0.5, cv::INTER_NEAREST);
cv::imshow("原图", dst1);
// 人像抠图
mat = detect_human(mat, downsample_ratio, session,
dynamic_input_node_dims,
dynamic_src_value_handler,
dynamic_r1i_value_handler,
dynamic_r2i_value_handler,
dynamic_r3i_value_handler,
dynamic_r4i_value_handler,
dynamic_dsr_value_handler,
input_node_names,
output_node_names,
context_is_update,
num_inputs,
num_outputs
);
// 人像抠图显示mask
/*mat = detect(mat,true, downsample_ratio, session,
dynamic_input_node_dims,
dynamic_src_value_handler,
dynamic_r1i_value_handler,
dynamic_r2i_value_handler,
dynamic_r3i_value_handler,
dynamic_r4i_value_handler,
dynamic_dsr_value_handler,
input_node_names,
output_node_names,
context_is_update,
num_inputs,
num_outputs
);*/
//结束时间，计算fps
double t2 = static_cast<double>(cv::getTickCount());
inferenceTime = (t2 - t1) / cv::getTickFrequency() * 1000;
std::stringstream fpsSs;
fpsSs << "FPS: " << int(1000.0f / inferenceTime * 100) / 100.0f;
// 向每帧写入fps
cv::putText(mat, fpsSs.str(), cv::Point(16, 32),
cv::FONT_HERSHEY_COMPLEX, 0.8, cv::Scalar(0, 0, 255));
//缩放显示
cv::resize(mat, dst2, cv::Size(), 0.5, 0.5, cv::INTER_NEAREST);
cv::imshow("抠图", dst2);
// 写入该帧到视频
if (outputVideo.isOpened())
outputVideo.write(mat);
cv::waitKey(1);
// 4. check context states.
if (!context_is_update) break;
if (_kbhit()) // 如果有按键被按下
{
if (_getch() == 'q') //如果按下了q键则跳出循环
{
break;
}
}
}
// 5. release
video_capture.release();
outputVideo.release();
destroyAllWindows();
}main.cpp
void showmenu()
{
printf("----菜单----\n");
printf("1-遍历文件夹抠图（保存对应抠图文件，默认选项）\n");
printf("2-图片抠图展示（不保存）\n");
printf("3-视频抠图展示(保存)\n");
printf("4-摄像头抠图展示（保存）\n");
printf("5-摄像头换背景展示（不保存）\n");
printf("0-退出函数\n");
}After generating the program, because my cuda path has several versions
Therefore, according to the missing dll prompt, copy all dlls under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1 to Release. The missing hdf5.dll is also found from the system (everything searches for one in V:\vcpkg-master\installed\x64-windows\bin, and there are other places too), copy it in.
OK, after this adjustment, it is no problem to double-click the generated exe without running it in the vs environment.
Reference URL:
1. https://github.com/caip1299920300/RobustVideoMatting_onnxruntime (source address)
2. https://github.com/microsoft/onnxruntime/releases/tag/v1.9.0
The End