Many friends will encounter examples where they need to use GPU in their work. GPU is the graphics card. Most of them have a vague concept of GPU, that is, it will be used in image rendering, but how it takes effect is unclear. This article mainly introduces how to use GPU for programming under opencv.
basic introduction
There are function interfaces that GPU can use under opencv, generally starting with cv::cuda:xxxx cv::cudacodec::, etc. cuda is a parallel computing framework provided by nvidia, which means that the cuda interface provided by opencv only supports nvidia's graphics cards (personal understanding). This article does not cover how to use other GPUs. So now that we have the nivida graphics card and the opencv library installed, can we directly use the gpu functions provided by opencv to operate? The answer is no. For nivida graphics cards, there is a nivida graphics card. Installing the driver is only a basic operation. You also need to download the corresponding cuda package from the nvidia official website for installation. You also need to download the Video_Codec_SDK_11.1.5.zip package to replace some header files. Only then. A detailed introduction will be given below.
Install cuda
- First, check the highest cuda version supported by the graphics card driver through the nvidia-smi command.
consys@consys-Lenovo-Legion-Y7000:~$ nvidia-smi
Mon Sep 25 19:14:41 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39 Driver Version: 460.39 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1050 Off | 00000000:01:00.0 Off | N/A |
| N/A 46C P8 N/A / N/A | 6MiB / 2000MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1214 G /usr/lib/xorg/Xorg 4MiB |
+--------------------------------------------------------------------------------------------------------------------------------- ----------------------------+
As shown in the above quote, the blue font part indicates the graphics card driver version number, and the red font part indicates the highest cuda version supported by the graphics card. The highest cuda toolkit version number supported by my own graphics card is 11.2.
2. After querying the highest cuda version number through the above command, go to the NVIDIA official website to download the cuda toolkit that meets the requirements. The screenshot of the official website is as follows:
Then you can download the corresponding version according to the characteristics of your own system. The selection method is as follows, everyone should know it, and I will not expand it here:
Correction here, just select runfile(local) in the picture above.
After downloading, the installation method is as follows:
sudo sh cuda_11.1.1_455.32.00_linux.run
After executing the above command, execute it step by step. It should be noted that if the driver has been installed on the GPU, there is no need to reinstall the driver. After the installation is complete, enter the nvcc -V command on the command line to check whether the cuda tool package is installed successfully.
consys@consys-Lenovo-Legion-Y7000:~/桌面$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:10:02_PDT_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.TC455_06.29069683_0
If the above prompt appears, it means that the cuda tool package is installed successfully. The installation path of the cuda tool package is /usr/local/cuda-11.1 .
3. After the cuda tool package is installed, it needs to be configured to take effect. The configuration method is as follows:
vi /etc/profileAfter opening the configuration file, add at the end:
export PATH=//usr/local/cuda-11.1/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-11.1/lib64:$LD_LIBRARY_PATHThe file takes effect through the source command, but the source command seems to only take effect in the terminal. After the terminal is closed, these two environment variables will not take effect. In order to take effect permanently, it is best to restart it.
At this point, the installation of the cuda tool package is complete.
Install cuDNN
cuDNN is suitable for deep learning, which will not be discussed here. Interested friends can search related articles on Baidu by themselves.
Install Video_Codec_SDK toolkit
Download address: Video Codec SDK - Get Started | NVIDIA Developer
After downloading, unzip the file, enter the directory and find the Read_Me.pdf file. You can find the system requirements of the sdk package, as shown in the following figure:
If the driver version is not satisfactory or the cuda toolkit is not satisfactory, the current lower version of nvidia-video-codec-sdk will suffice.
Why should you install this SDK toolkit?
In fact, during the GPU driver installation process, the nvidai-video-codec-sdk library file has been installed. It is generally installed in the /usr/lib/x86_64-linux-gnu/ directory, such as the 525.89.02 version of the GPU driver installation. After that, the library files of libnvcuvid.so.525.89.02 and libnvidia-encode.so.525.89.02 exist in the /usr/lib/x86_64-linux-gnu/ directory.
Therefore, only the header file is needed. Copy the header file to the cuda toolkit directory through the following command:
cp Video_Codec_SDK_11.1.5/Interface/* /usr/local/cuda/include/Note: The above only uses the header files in nvidia-video-codec-sdk, but does not use the libnvcuvid.so and libnvidia-encode.so libraries in nvidia-video-codec-sdk. The reason is that when installing the graphics card driver, By default, libnvcuvid.so and libnvidia-encode.so are installed that are compatible with the driver version, and the library in nvidia-video-codec-sdk is likely to be inconsistent with the graphics card driver version we installed. If nvidia-video-codec-sdk is used There may be no problem when compiling libnvcuvid.so and ibnvidia-encode.so, but when running, an error is likely to be reported due to incompatibility with the driver version, because libnvcuvid in nvidia-video-codec-sdk is refused to be used. so, ibnvidia-encode.so library. This can be said to be Nvidia’s sinkhole, so be sure to pay attention.
Without this step, although no errors will occur when compiling opencv, an error will be reported when compiling and connecting programs compiled by GPU. The error message is as follows: function/feature is not implement, the called functionality is disabled for current build or platform in function 'throw_no_cuda' .
Reference link: [AVD] Linux compiles OpenCV 4.6 that supports Cuda, solving the error throw_no_cuda_Deep Sea Enoch's Blog-CSDN Blog
Install ffmpeg
What is the role of ffpmeg in opencv? Opencv relies on ffmpeg for soft decoding and nvidia-video-codec-sdk for hard decoding . If ffmpeg is not installed, opencv cannot perform soft decoding. My understanding is that it cannot read video files or stream playback. Therefore, in order to ensure that opencv can decode softly and hard, both ffmpeg and nvidia-video-codec-sdk need to be installed. The installation of nvidia-video-codec-sdk has been introduced in the previous section. This session will introduce how to install ffmpeg. It should be noted that ffmpeg also provides a way to use NVIDIA for hard decoding, so ffmpeg can also use NVIDIA graphics cards for hard decoding .
Install dependencies for ffmpeg
sudo apt update
sudo apt install autoconf \
automake \
build-essential \
cmake \
git-core \
libass-dev \
libfreetype6-dev \
libgnutls28-dev \
libsdl2-dev \
libtool \
libva-dev \
libvdpau-dev \
libvorbis-dev \
libxcb1-dev \
libxcb-shm0-dev \
libxcb-xfixes0-dev \
pkg-config \
texinfo \
wget \
yasm \
zlib1g-dev
If you want to compile ffmpeg nvidia hard decoding in docker, you need to copy the libnvcuvid.so and libnvidia-encode.so libraries installed when installing the graphics card driver from the host to docker. The paths of these two libraries on the host are generally In the /usr/lib/x86_64-linux-gnu/ directory, you can copy the above two libraries to docker in advance, and then copy them to the /lib64 directory of docker (be sure to copy from the host directory, do not use Video_Codec_SDK The libraries in Video_Codec_SDK may not match the drivers installed on this machine. Even if the compilation passes, driver incompatibility problems will occur during runtime). For example, the two libraries are libnvcuvid.so.525.89.02 and libnvidia- encode.so.525.89.02, the operation in docker is as follows:
cp libnvcuvid.so.525.89.02 /lib64/
cp libnvidia-encode.so.525.89.02 /lib64/
ln -s /lib64/libnvcuvid.so.525.89.02 /lib64/libnvcuvid.so.1
ln -s /lib64/libnvidia-encode.so.525.89.02 /lib64/libnvidia-encode.so.1
echo '/lib64' >> /etc/ld.so.conf
ldconfig
Just go to github and download the corresponding version of ffmpeg.
It is particularly important to note that ffmpeg provides header files that can operate NVIDIA graphics cards. The download command is as follows:
git clone https://git.videolan.org/git/ffmpeg/nv-codec-headers.gitThe directory after the download is completed is nv-codec-headers, and the screenshot is as follows:
From the above, you can see that there are some header files in the include directory, and the header files are some functions that can operate the cuda toolkit. From this, we can understand that ffmpeg uses the api provided by cuda to control the gpu.
What needs special attention is that the dependencies of nv-codec-headers are described in the README file in the above figure:
nv-codec-headers installation method:
cd nv-codec-headers && sudo make installThe script to compile ffmpeg is as follows:
#!/bin/bash
./configure --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --disable-static --enable-shared
make -j$(nproc)
sudo make install
echo '/usr/local/ffmpeg/lib' >> /etc/ld.so.conf
ldconfigAfter the above execution is completed, test ffmpeg hardware access and cuvid decoder:
ffmpeg -hwaccels
ffmpeg -codecs | grep cuvidThe test results are as follows:
consys@consys-Lenovo-Legion-Y7000:~$ ffmpeg -hwaccels
ffmpeg version 4.2.2-1kylin1k21.5 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9 (Ubuntu 9.3.0-10kylin2)
configuration: --prefix=/usr --extra-version=1kylin1k21.5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
WARNING: library configuration mismatch
avcodec configuration: --prefix=/usr --extra-version=1kylin1k21.5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libaribb24 --enable-liblensfun --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Hardware acceleration methods:
vdpau
cuda
vaapi
drm
opencl
cuvid
The ffmepg test hard decoding method is as follows:
ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -extra_hw_frames 5 -i out.mp4 -c:a copy -c:v h264_nvenc -b:v 5M out1.mp4I'm not sure about some of the above options, so interested friends can do their own research. The above command inputs an mp4 file and then outputs an mp4 file.
Recompile opencv
After completing the above steps, you need to recompile opencv through the cmke-gui command, because opencv does not turn on cuda related options when compiling by default. Opencv has many compilation options, some of which are related to cuda.
The interface for cmake to open opencv is as follows. I filtered it through the keyword cuda:
You can see WITH_CUDA from the picture above . This option must be checked; the OPENCV_DNN_CUDA option can be checked if cuDNN is installed; confirm whether the path of CUDA_TOOLKIT_ROOT_DIR is correct. If it is inconsistent with the installation path, change it to the installation path of the cuda toolkit; CUDA_ARCH_BIN configuration Method: Search on Baidu. The other check options are the same as above.
There is also the WITH_FFMPEG option, which must be checked.
For the compilation method of opencv under Linux, please refer to: How to install OpenCV under Linux - Zhihu
Follow the above connection steps and click the Configure button on cmake-gui. You can see the configuration information of some dependencies under opencv. Here we focus on the configuration information of ffmpeg and NVIDIA. Others are similar, as shown below:
Video I/O:
DC1394: YES (2.2.5)
FFMPEG: YES
avcodec: YES (58.54.100)
avformat: YES (58.29.100)
avutil: YES (56.31.100)
swscale: YES (5.5.100)
resample: NO
GStreamer: YES (1.16.2)
v4l/v4l2: YES (linux/videodev2.h)
NVIDIA CUDA: YES (ver 11.1, CUFFT CUBLAS NVCUVID FAST_MATH)
NVIDIA GPU arch: 35 37 50 52 60 61 70 75 80 86
NVIDIA PTX archs:
cuDNN: YES (ver 8.9.2)
The above NVCUVID option must be present, otherwise opencv cannot use the GPU for hard decoding and an error will be reported. The error message here is: function/feature is not implement, the called functionality is disabled for current build or platform in function 'throw_no_cuda' .
Regarding some errors under docker, you can refer to the following information. I copied it from elsewhere and it has not been verified. Friends need to verify it themselves:
Consider whether libnvcuvid.so and libnvidia-encode.so are not appended to the search path of the library. This problem generally occurs in the docker environment. You can copy it from the host /usr/lib/x86_64-linux-gnu directory. The above libraries (must be copied from the host directory, do not use the libraries in Video_Codec_SDK, because the libraries in Video_Codec_SDK are likely to not match the drivers installed on the machine. Even if the compilation passes, driver incompatibility will occur during runtime. ), for example, libnvcuvid.so.525.89.02, libnvidia-encode.so.525.89.02 are copied to the /usr/lib/x86_64-linux-gnu directory in docker, and a soft connection is created. The soft connection script is as follows:
#!/bin/bash
sopath=/usr/lib/x86_64-linux-gnu
if [ ! -L ${sopath}/libcuda.so ]; then
files=(`find $sopath/libcuda.so*`)
raw_so=${files[0]}
echo Create soft link ${raw_so}
ln -s ${raw_so} ${sopath}/libcuda.so
fi
if [ ! -L ${sopath}/libnvcuvid.so ]; then
echo Create soft link ${sopath}/libnvcuvid.so.1
ln -s ${sopath}/libnvcuvid.so.1 ${sopath}/libnvcuvid.so
fi
if [ ! -L ${sopath}/libnvidia-encode.so ]; then
echo Create soft link ${sopath}/libnvidia-encode.so.1
ln -s ${sopath}/libnvidia-encode.so.1 ${sopath}/libnvidia-encode.so
fi
ldconfigThen click gennerate in cmake-gui to trigger the generation of makefile files. and then execute
make -j$(nproc)
make installopencv hard decoding test code:
void MainWindow::playVideoByGpu()
{
const std::string fname("rtsp://admin:[email protected]//Streaming/Channels/1");
//cv::cuda::setGlDevice();
cv::cuda::GpuMat d_frame;
cv::cuda::GpuMat d_outFrame;
cuda::Stream stream;
cv::Ptr<cv::cudacodec::VideoReader> d_reader = cv::cudacodec::createVideoReader(fname);
while(1)
{
if (!d_reader->nextFrame(d_frame))
break;
cuda::resize(d_frame,d_outFrame,Size(1920,1080));
cv::Mat temp;
d_outFrame.download(temp, stream);
stream.waitForCompletion();
cv::imshow("GPU", temp);
if (cv::waitKey(30) > 0)
break;
}
}But something strange is that in my environment, I used the GPU to perform hard decoding to read the video, and then called imshow on the main thread to display the video. I did not find that the CPU occupancy rate dropped significantly. Friends who know the reason can leave me a message. Thank you so much!
Reference connection: