It is used to record some problems and solutions encountered in the process of Intel CPU developing qsv hardware decoding
The following articles are more meaningful articles for reference during the development process, for your study and reference~~
https://zhuanlan.zhihu.com/p/62246545
##FFMPEG+Intel QSV hard solution environment installation articles##
https://zhuanlan.zhihu.com/p/372361709
##Ubuntu20.04 ffmpeg adds Intel core display QSV acceleration support##
https://blog.csdn.net/weixin_47407737/article/details/128933104
##FFmpeg integrated qsv compilation and installation##
https://pic.huodongjia.com/ganhuodocs/2018-05-26/ 1527317476.4.pdf
## FFmpeg hardware acceleration and optimization on Intel GPU ##
https://blog.csdn.net/kkae8643150/article/details/107472572
##ubuntu18.0.4 compile ffmpeg open qsv hardware codec ##
| describe | git warehouse link address |
|---|---|
| Driver dependent library | https://github.com/intel/libva |
| Driver Test Tool | https://github.com/intel/libva-utils |
| Driver dependent library | https://github.com/intel/gmmlib |
| Driver | https://github.com/intel/media-driver/tree/intel-media-21.3 |
| Driver | https://github.com/intel/intel-vaapi-driver |
| MediaSDK | https://github.com/Intel-Media-SDK/MediaSDK/wiki/Build-FFmpeg-QSV |
| MediaSDK | https://github.com/Intel-Media-SDK/MediaSDK/wiki/Build-Media-SDK-on-Ubuntu |
| MediaSDK | https://github.com/Intel-Media-SDK/MediaSDK/wiki/Intel-media-stack-on-Ubuntu |
| MediaSDK | https://github.com/Intel-Media-SDK/MediaSDK/releases |
| FFMPEG | https://github.com/FFmpeg/FFmpeg |
| RUN TIME | https://github.com/oneapi-src/oneVPL-intel-gpu |
intel-media-driver+intel-media-sdk = iHD(i915 driver)
vaapi-driver = i965 driver
Preparation
install dependencies
2222 sudo apt-get install git cmake pkg-config meson libdrm-dev automake libtool
gcc/g++ cmake version requirements
There are version requirements for gcc and cmake during the compilation process. You need to upgrade gcc and cmake to a certain version first. The version requirements of gcc/g++ are greater than or equal to 4.9, and the version requirements of cmake are greater than or equal to 3.6.
important point
In addition, before starting the compilation work, you need to pay attention to the corresponding relationship between the versions of libva/media-driver and Media-SDK, otherwise there will be a compilation error. The release document of Intel-Media-SDK (link: Intel Media SDK to know which tag of libva and media-driver is used to compile.
test environment:
A total of four x86 machines were tested
| CPU model | CPU architecture |
|---|---|
| Intel J1900 | Bay Trail Gen7 |
| Intel i5-6442EQ | Skylake Gen9 |
| Intel i7-7600U | Kaby Lake Gen9.5 |
| Intel i7-8700 | Coffee Lake Gen9.5 |
Among them, the Intel i7-8700 machine has both independent display and integrated display. In order to avoid the need to explicitly specify -qsv_device or -hwaccel_device during debugging, you can enter the BIOS after booting and set the default to use integrated display or use independent display or automatic Select, the default setting of the system is AUTO.
#用来查看当前显示的配置
2256 lspci | grep -i vga
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Oland [Radeon HD 8570 / R7 240/340 OEM] (rev 87)
2257 cat /sys/kernel/debug/dri/128/name
2258 cat /sys/kernel/debug/dri/129/name
Among them, /dev/dri/renderD128 corresponds to integrated display, and /dev/dri/renderD129 corresponds to independent display.
ubuntu16.04 upgrade gcc/g++
69 add-apt-repository ppa:ubuntu-toolchain-r/test
70 apt-get update
88 apt-get install gcc-8 g++-8
91 gcc -v
95 update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 20
96 update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 20
97 update-alternatives --config gcc
98 update-alternatives --config g++
ubuntu16.04 upgrade cmake
https://askubuntu.com/questions/829310/how-to-upgrade-cmake-in-ubuntu
2203 chmod +x cmake-3.25.2-linux-x86_64.sh
2204 ./cmake-3.25.2-linux-x86_64.sh
2212 cd /home/ning/Downloads/cmake-3.25.2-linux-x86_64/bin/
2213 ./cmake --version
2214 ln -sf /home/ning/Downloads/cmake-3.25.2-linux-x86_64/bin/cmake /usr/local/bin/cmake
2215 cd /usr/local/bin/
2216 ./cmake --version
install libva
2224 git clone https://github.com/intel/libva.git
2226 cd libva/
2227 git checkout v2.13-branch
2228 ./autogen.sh --prefix=/opt/intel/mediasdk/
2229 make -j8
2231 make install
After the libva installation is complete, you need to configure the environment variable export LIBVA_DRIVER_NAME=iHD
Install va-utils
2247 git clone https://github.com/intel/libva-utils.git
2248 git checkout v2.13-branch
2249 cd libva-utils/
2250 git checkout v2.13-branch
2251 ./autogen.sh --prefix=/opt/intel/mediasdk
2252 make -j8
2253 make install
install gmmlib
2234 tar xf gmmlib-intel-gmmlib-21.3.1.tar.gz
2235 cd gmmlib-intel-gmmlib-21.3.1
或者
git clone https://github.com/intel/gmmlib.git
cd gmmlib
git reset --hard 1c975c52634172376b49f2983d4d0fd7049302f9
2236 mkdir build
2237 cd build/
2238 cmake -DCMAKE_BUILD_TYPE=Release -DARCH=64 -DCMAKE_INSTALL_PREFIX=/opt/intel/mediasdk ..
2239 make -j8
2240 make install
Install media-driver
2242 git clone https://github.com/intel/media-driver.git
2243 cd media-driver/
2244 git checkout intel-media-21.3
2245 git br
2246 cd ../
2265 cd media-driver/
2266 git br
2267 cd ../
2268 mkdir build_media
2269 cd build_media/
2270 cmake -DCMAKE_INSTALL_PREFIX=/opt/intel/mediasdk/ ../media-driver/
2271 cd ../media-driver/
2272 git br
2273 git log
2274 git reset --hard 46458db8104e70602e99bd233119d022bc54a39a
2275 cd ../
2276 cd build_media/
2277 rm -rf ./*
2278 cmake -DCMAKE_INSTALL_PREFIX=/opt/intel/mediasdk/ ../media-driver/
2279 make -j8
2280 make install
After the driver installation is complete, you need to configure the environment variable export LIBVA_DRIVERS_PATH=/opt/intel/mediasdk/lib/dri/;
after the above environment variables are configured, you can use the function vainfo generated by va-utils to verify whether the driver installation is successful. Read the value of the environment variable to determine whether the environment variable is set correctly;
2307 echo $LIBVA_DRIVERS_PATH
2308 echo $LIBVA_DRIVER_NAME
2309 #这里，media-driver对应的LIBVA_DRIVERS_NAME=iHD，$LIBVA_DRIVERS_PATH=iHD_drv_video.so
Test with vainfo
#由于测试机器i7-8700同时存在集显和独显，所以这里需要指定--display drm --device，如果只有集显，执行运行vainfo即可
2278 ./vainfo
或者
2279 ./vainfo --display drm --device /dev/dri/renderD128
2279 root@ning-QiTianM620-N000:/opt/intel/mediasdk/bin# ./vainfo
2279 error: XDG_RUNTIME_DIR not set in the environment.
2279 libva info: VA-API version 1.13.0
2279 libva info: User environment variable requested driver 'i965'
2279 libva info: Trying to open /opt/intel/mediasdk/lib/dri/i965_drv_video.so
2279 libva info: Found init function __vaDriverInit_1_13
2279 libva info: va_openDriver() returns 0
2279 vainfo: VA-API version: 1.13 (libva 2.13.0)
2279 vainfo: Driver version: Intel i965 driver for Intel(R) Coffee Lake - 2.4.1
2279 vainfo: Supported profile and entrypoints
2279 VAProfileMPEG2Simple : VAEntrypointVLD
2279 VAProfileMPEG2Simple : VAEntrypointEncSlice
2279 VAProfileMPEG2Main : VAEntrypointVLD
2279 VAProfileMPEG2Main : VAEntrypointEncSlice
2279 VAProfileH264ConstrainedBaseline: VAEntrypointVLD
2279 VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
2279 VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
2279 VAProfileH264Main : VAEntrypointVLD
2279 VAProfileH264Main : VAEntrypointEncSlice
2279 VAProfileH264Main : VAEntrypointEncSliceLP
2279 VAProfileH264High : VAEntrypointVLD
2279 VAProfileH264High : VAEntrypointEncSlice
2279 VAProfileH264High : VAEntrypointEncSliceLP
2279 VAProfileH264MultiviewHigh : VAEntrypointVLD
2279 VAProfileH264MultiviewHigh : VAEntrypointEncSlice
2279 VAProfileH264StereoHigh : VAEntrypointVLD
2279 VAProfileH264StereoHigh : VAEntrypointEncSlice
2279 VAProfileVC1Simple : VAEntrypointVLD
2279 VAProfileVC1Main : VAEntrypointVLD
2279 VAProfileVC1Advanced : VAEntrypointVLD
2279 VAProfileNone : VAEntrypointVideoProc
2279 VAProfileJPEGBaseline : VAEntrypointVLD
2279 VAProfileJPEGBaseline : VAEntrypointEncPicture
2279 VAProfileVP8Version0_3 : VAEntrypointVLD
2279 VAProfileVP8Version0_3 : VAEntrypointEncSlice
2279 VAProfileHEVCMain : VAEntrypointVLD
2279 VAProfileHEVCMain : VAEntrypointEncSlice
2279 VAProfileHEVCMain10 : VAEntrypointVLD
2279 VAProfileHEVCMain10 : VAEntrypointEncSlice
2279 VAProfileVP9Profile0 : VAEntrypointVLD
2279 VAProfileVP9Profile0 : VAEntrypointEncSlice
2279 VAProfileVP9Profile2 : VAEntrypointVLD
2279 root@ning-QiTianM620-N000:/opt/intel/mediasdk/bin#
Install Intel-Media-SDK
2315 git clone https://github.com/Intel-Media-SDK/MediaSDK.git
2316 cd MediaSDK/
2317 git checkout intel-mediasdk-21.3
2318 mkdir build
2319 cd build
2320 cmake -DCMAKE_INSTALL_PREFIX=/opt/intel/mediasdk ..
2321 make -j8
2322 make install
It should be noted here that if the library or header file cannot be found when compiling the SDK, you need to export to set the environment variable
export PKG_CONFIG_PATH=/opt/intel/mediasdk/lib/pkgconfig
In this directory, many .pc files are stored, and the reference paths of libraries and header files are specified.
install ffmpeg
ffmpeg installed via apt-get itself does not support the QSV option, so you must recompile it yourself.
When using Intel-Media-SDK, that is, Driver uses iHD_dri_video.so, when configuring ffmpeg, you need to add the –enable-libmfx option; when
using vaapi, that is, Driver uses i965_dri_video.so, when configuring ffmpeg, you need to add –enable-vaapi option;
In addition, Compile If you need to generate ffplay to play video, you also need to add the --enable-ffplay option to configure ffmpeg.
2327 git clone https://github.com/ffmpeg/ffmpeg.git
2330 cd ffmpeg/
2331 git br
2332 git checkout release/3.4
2333 ./configure --arch=x86_64 --prefix=/opt/intel/mediasdk/ --disable-yasm --enable-vaapi --enable-libmfx --enable-debug=3 --disable-stripping --extra-cflags=-gstabs+ --disable-optimizations
2334 make -j8
2335 make install
Install vaapi-driver
The reason why the iHD Driver cannot be used on the Intel J1900 machine is described in the figure:
Therefore, on the J1900 machine, the vaapi driver must be used to realize the hardware decoding function.
560 git clone https://github.com/intel/intel-vaapi-driver.git
561 cd intel-vaapi-driver/
562 git checkout v2.4-branch
563 git branch
564 ls -ll
565 ./autogen.sh --prefix=/opt/intel/mediasdk/
566 make -j4
567 make install
568 #这里，media-driver对应的LIBVA_DRIVERS_NAME=i965，$LIBVA_DRIVERS_PATH=i965_drv_video.so
Test hardware decoding with ffmpeg
#1080P 同时存在集显和独显的情况下，使用-hwaccel_device指定集显设备
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device /dev/dri/renderD128 -i RISC_e.mp4 -f null -
#4K视频
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i 3840x2160.mp4 -vf 'hwdownload,format=nv12' -f null -
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i 3840x2160.mp4 -f null -
#1080P
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i RISC_e.mp4 -vf 'hwdownload,format=nv12' -f null -
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i RISC_e.mp4 -vf -f null -
Note that the difference in the above instructions is the use of the filter -vf 'hwdownload, format=nv12'
Use 'hwdownload' to download the gpu frame output by hardware decoding to cpu memory, the keyword is to download and
use 'format=NV12' to select NV12 The data in the format is used as the output of hwdownload
-f null - to output the gpu frame output by hardware decoding to the cpu memory without saving it, and do not add -vf 'hwdownload,format=nv12' for 4K video. The error is suspected to be caused by insufficient cpu memory. Using hwdownload will not report an error
install ffplay
ffplay compilation mainly depends on the sdl2 environment (for ffmpeg 3.x version), if sdl2 is not installed, the compilation options for ffplay will not be automatically generated
sudo apt-get install libsdl2-2.0
# libSDL2-2.0.so.0位于/usr/lib/x86_64-linux-gnu/目录下
#ffmpeg configure时加上 --enable-ffplay选项即可
2333 ./configure --arch=x86_64 --prefix=/opt/intel/mediasdk/ --disable-yasm --enable-vaapi --enable-libmfx --enable-debug=3 --disable-stripping --extra-cflags=-gstabs+ --disable-optimizations --enable-ffplay
In addition, ffplay depends on the VPL runtime environment. Users can set the following environment variables to specify VPL Runtime or MSDK Runtime (default)
export INTEL_MEDIA_RUNTIME=ONEVPL #for VPL Runtime:libmfx-gen.so.1.2
或者
export INTEL_MEDIA_RUNTIME=MSDK #for Media SDK Runtime:libmfxhw64.so.1
Install libmfx-gen.so.1.2, if you do not install ffplay to play the video will report an error
https://bugs.launchpad.net/ubuntu/+source/ffmpeg/+bug/1970637
2085 git clone https://github.com/oneapi-src/oneVPL-intel-gpu.git
2088 cd oneVPL-intel-gpu/
2089 git br
2090 git checkout intel-onevpl-21.2
2091 git br
2092 mkdir build
2093 cd build
2094 cmake ..
2095 make -j8
2096 make install
2097 cd /opt/intel/mediasdk/lib/
2098 ls （libmfx-gen*.so）
2101 export INTEL_MEDIA_RUNTIME=ONEVPL（这步一定要加上）
2099 cd ../bin/
2102 ./ffplay -vcodec h264_qsv -i RISC_e.mp4
2103 ./ffplay -vcodec h264_qsv -i /home/ning/Downloads/3840x2160.mp4
VAAPI driver (hardware decoding + hardware encoding + saving yuv files)
Need to explain here, why do you want to save the yuv file?
The reason is that when using the vaapi driver, ffplay cannot specify the decoder through -vcodec, so it is impossible to use ffplay to directly decode and play video through hardware, so as to reduce the CPU usage of the device. Therefore, saving the yuv file, that is to say, first complete the hardware decoding output, and then use ffplay to play it, can reduce the CPU usage of the device.
#硬件解码+硬件编码
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i RISC_e.mp4 -c:v h264_vaapi /home/ning/Downloads/RISC_vaapi.mp4
#硬件解码+输出yuv文件
#YUV420P的片源通常會硬體加速解碼會輸出NV12格式
./ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i RISC_e.mp4 -vf 'hwdownload,format=nv12' -pix_fmt yuv420p /home/ning/Downloads/risc_e.yuv
#ffplay播放yuv文件
./ffplay -pix_fmt yuv420p -f rawvideo -video_size 1920x1080 /home/ning/Downloads/risc_e.yuv
#这里注意需要填相应的分辨率
#16M MP4文件大小，转换为yuv后为8.5G
iHD driver (hardware decoding + hardware encoding + saving yuv files)
#硬件解码+硬件编码
./ffmpeg -hwaccel qsv -c:v h264_qsv -i RISC_e.mp4 -c:v h264_qsv /home/ning/Downloads/RISC_qsv.mp4
#硬件解码+输出yuv文件
#YUV420P的片源通常會硬體加速解碼會輸出NV12格式
./ffmpeg -hwaccel qsv -c:v h264_qsv -i RISC_e.mp4 -vf 'hwdownload,format=nv12' -pix_fmt yuv420p /home/ning/Downloads/risc_e_qsv.yuv
#ffplay播放MP4，使用硬件解码
./ffplay -vcodec h264_qsv -i /home/ning/Downloads/RISC_vaapi.mp4
Use ffmpeg to capture a length of video
-codec copy means to use the same decoder as the input video, and not to re-encode
./ffmpeg -i 3840x2160.mp4 -vcodec copy -acodec copy -ss 00:00:00 -to 00:00:10 4k_test.mp4 -y
The reason for intercepting a piece of MP4 video here is because the 4K resolution video has too much data after hardware decoding from MP4 format to yuv format, so a small section of it needs to be intercepted for testing.