Qt audio and video development 21-general hard decoding

I. Introduction

Hardware decoding is a scheme proposed by graphics chip manufacturers to decode video streams with GPU resources. The opposite is soft decoding, which is the traditional scheme of using CPU to undertake decoding work; the advantages are high efficiency, low power consumption, and low thermal power consumption. , The disadvantage is the lack of strong support (including filters, subtitles, etc.), large limitations (such as the PC’s energy-saving function failure cnq after turning on the hardware decoding, etc.), and the settings are more complicated; hardware decoding modules and related Driver coordination, appropriate playback software, and correct settings for the playback software, the hardware decoding function cannot be turned on without one. The mainstream hardware decoding solutions are introduced by Intel, AMD-ATI and Nvdia.

At present, mainstream players all support hard decoding. Players without hard decoding are rogues. There are more and more 2K 4K or even 8K HD videos. If there is no hard decoding and only soft decoding, it will consume a lot of CPU resources, which is very unacceptable. It needs to be specially handed over to the GPU for drawing and decoding, which can greatly reduce the pressure on the CPU and allow the CPU to do other tasks. In the video surveillance system I wrote before, we used excellent decoding and GPU drawing, and did not use these two Everything is just empty talk. It can’t bear 64-channel real-time display at all. VLC, ffmpeg, and mpv definitely support hard decoding. The settings of vlc and mpv are the simplest. Haikang’s SDK does not provide hard decoding by default. There is a library of hard decoding written in the manual, it is estimated that you need to contact to obtain it.

Hard decoding processing of different cores

1. Vlc hard decoding:
   1.1 Set parameters by calling libvlc_media_add_option: avcodec-hw=auto
   1.2 Supported parameters: none auto any d3d11va dxva2

2. Hard decoding ffmpeg:
   2.1 av_hwdevice_find_type_by_name calling name automatically find hard decoding according to
   2.2 calls find_fmt_by_hw_type hard to find the corresponding decoding scheme
   2.3 callback videoCtx-> get_format = get_hw_format decoder hardware decoding format assigned to
   2.4 hardware decoding apparatus create call av_hwdevice_ctx_create
   2.5 decoded data call avcodec_send_packet avcodec_receive_frame
   2.6 Call av_hwframe_transfer_data to convert hard-decoded data
   2.7 Supported parameters: none qsv cuvid d3d11va dxva2

3. mpv hard decoding:
   3.1 Set the parameter hwdec by calling mpv_set_option
   3.2 Supported parameters: none auto any d3d11va dxva2

2. Features

1. Support multi-screen switching, full-screen switching, etc., including 1+4+6+8+9+13+16+25+36+64 screen switching.
2. Support alt+enter full screen, esc exit full screen.
3. Custom information box + error box + inquiry box + prompt box in the lower right corner (including multiple formats).
4. 17 sets of skin styles can be changed at will, all styles are unified, including menus.
5. Move the mouse on the gimbal dashboard to highlight, and the eight directions are accurately identified.
6. The bottom screen toolbar (screen split switch + screenshot sound and other settings) is moved up and highlighted.
7. The logo+Chinese software name+English software name can be changed in the configuration file.
8. Encapsulates Baidu map, view switching, motion track, device position, and mouse click to get latitude and longitude, etc.
9. Support picture map, the device button can be dragged freely on the picture map to automatically save location information.
10. On Baidu map and picture map, double-click the video to preview the camera real-time video.
11. Stack window, each window is a separate qwidget, convenient to write your own code.
12. The right-click menu of the top mouse can dynamically control the display and hide of the time CPU + upper left panel + lower left panel + upper right panel + lower right panel, and supports restoring the default layout.
13. The toolbar can place multiple small icons and close icons.
14. The left and right sides can be dragged and stretched, and the width and height positions are automatically memorized, and will be restored after restart.
15. Double-click the camera node to automatically play the video, double-click the node to automatically add videos in sequence, and it will automatically skip to the next one. Double-click the parent node to automatically add all the videos under the node.
16. The camera node is dragged to the corresponding window to play the video, and local files are supported for direct playback.
17. The video frame supports dragging and swapping, and it responds instantly.
18. Double-click the node + drag the node + drag the form to swap positions, and url.txt will be updated automatically.
19. Support loading channel video playback from url.txt, automatically remember the video corresponding to the last channel, and automatically open and play after the software is started.
20. The volume bar control in the lower right corner, automatically hides when the focus is lost, and the volume bar has a mute icon.
21. Integrated Baidu online map and offline map, you can add the corresponding location of the device, automatically generate a map, support zooming and adding overlays, etc.
22. Drag the video outside the channel window to automatically delete the video.
23. Right mouse button can delete current+all videos, screenshot current+all videos.
24. Recorder management, camera management, you can add, delete, modify, import and export print information, and immediately apply the new device information to generate a tree list without restarting.
25. You can freely enable whether to load the map in the pro file.
26. Two kinds of kernels can be switched freely for video playback, vlc+ffmpeg, both can be set in pro.
27. 1+4+9+16 screen polling can be set, polling interval and polling stream type can be set, just click the start polling button on the right side of the toolbar at the bottom of the main interface, and click again to stop polling .
28. By default, the mouse pointer will be automatically hidden without operation for more than 10 seconds.
29. Support onvif search equipment, support any onvif camera, including but not limited to Haikang Dahuayu Shitiandi Weiye Huawei, etc.
30. Support onvif PTZ control, can move PTZ camera up, down, left and right, including reset and focus adjustment.
31. At the same time, it supports databases such as sqlite, mysql, and postsql.
32. Video can be saved, timing storage or single file storage is optional, and storage interval is optional.
33. You can set the video stream communication method tcp+udp, and you can set the video decoding to be speed priority, quality priority, equalization, etc.
34. Can set the hard decoding type, support qsv, dxva2, d3d11va, etc.
35. By default, opengl is used to draw video, which has ultra-low CPU resource consumption and supports yuyv and nv12 format drawing, which is very awesome.
36. Highly customizable, users can easily derive their own functions on this basis, supporting linux and mac systems.

Three, renderings

Four, related sites

1. Domestic site: https://gitee.com/feiyangqingyun/QWidgetDemo
2. International site: https://github.com/feiyangqingyun/QWidgetDemo
3. Personal homepage: https://blog.csdn.net/feiyangqingyun
4. Zhihu Homepage: https://www.zhihu.com/people/feiyangqingyun/
5. Experience address: https://blog.csdn.net/feiyangqingyun/article/details/97565652

Five, the core code

bool FFmpegThread::initHWDeviceOther()
{
#ifdef hardwarespeed
    //根据名称自动寻找硬解码
    QByteArray hardwareData = hardware.toUtf8();
    enum AVHWDeviceType type = av_hwdevice_find_type_by_name(hardwareData.data());
    qDebug() << TIMEMS << "AVHWDeviceType" << type;

    //找到对应的硬解码格式
    hw_pix_fmt = find_fmt_by_hw_type(type);
    if (hw_pix_fmt == -1) {
        qDebug() << TIMEMS << "cannot support hardware";
        return false;
    }

    AVStream *videoStream = formatCtx->streams[videoStreamIndex];
    videoCtx = avcodec_alloc_context3(videoCodec);
    if (!videoCtx) {
        qDebug() << TIMEMS << "avcodec_alloc_context3 error";
        return false;
    }

    int result = -1;
    result = avcodec_parameters_to_context(videoCtx, videoStream->codecpar);
    if (result < 0) {
        qDebug() << TIMEMS << "avcodec_parameters_to_context error";
        return false;
    }

    //解码器格式赋值为硬解码
    videoCtx->get_format = get_hw_format;
    //av_opt_set_int(videoCtx, "refcounted_frames", 1, 0);

    //创建硬解码设备
    AVBufferRef *hw_device_ref;
    result = av_hwdevice_ctx_create(&hw_device_ref, type, NULL, NULL, 0);
    if (result < 0) {
        qDebug() << TIMEMS << "open the hardware device error";
        return false;
    }

    videoCtx->hw_device_ctx = av_buffer_ref(hw_device_ref);
    av_buffer_unref(&hw_device_ref);
#endif
    return true;
}