Qt audio and video development 40-ffmpeg capture desktop and record

I. Introduction

I used ffmpeg to get through the collection of various video files, video streams and local camera equipment. Recently, a customer requested to capture the entire desktop screen and record and save it as an MP4 file. I have encountered similar needs before. Because I haven't done it before, and I don't have the energy to explore and test it, so I have been delaying it. Recently, this demand just came again. I settled down and tried it. It turned out to be very simple. The mode is the same as that of collecting local camera equipment, calling the parameters of av_find_input_format Just change from dshow to gdigrab, and then change the open address to desktop, and the whole process of saving to MP4 files after collection is exactly the same, no need to make any changes at all, it is completely usable, it can be said that ffmpeg is packaged very well and considered very well Thoughtful, the design of various function interfaces is also very beautiful.

Before implementing this function, the first step is to use the command line to test, for example, ffplay -f gdigrab desktop -video_size 640x480 -framerate 30 means to enable desktop capture, starting from the coordinate 00 in the upper left corner, a rectangular area with a resolution of 640x480, according to the frame rate 30 playback, only when the command line is available, it means that the ffmpeg code works. If the command line is not available, then carefully think about where the problem is, whether it is a problem with the command line itself or the device does not support it. Including ffmpeg to implement the streaming function is also the same idea. Try to use the command line first to see if it works. Only when it is feasible can you boldly implement it with code, which is equivalent to exploring with the command line first.

Different operating systems correspond to different device collection format names, which need to be handled differently.

void FFmpegThread::initInputFormat()
{
    
    
    //本地摄像头/桌面录屏
    if (videoType == VideoType_Camera) {
    
    
#if defined(Q_OS_WIN)
        //ifmt = av_find_input_format("vfwcap");
        ifmt = av_find_input_format("dshow");
#elif defined(Q_OS_LINUX)
        //可以打开cheese程序查看本地摄像头(如果是在虚拟机中需要设置usb选项3.1)
        //ifmt = av_find_input_format("v4l2");
        ifmt = av_find_input_format("video4linux2");
#elif defined(Q_OS_MAC)
        ifmt = av_find_input_format("avfoundation");
#endif
    } else if (videoType == VideoType_Desktop) {
    
    
#if defined(Q_OS_WIN)
        ifmt = av_find_input_format("gdigrab");
#elif defined(Q_OS_LINUX)
        ifmt = av_find_input_format("x11grab");
#elif defined(Q_OS_MAC)
        ifmt = av_find_input_format("avfoundation");
#endif
    }
}

2. Rendering

3. Experience address

1. Domestic site: https://gitee.com/feiyangqingyun
2. International site: https://github.com/feiyangqingyun
3. Personal works: https://blog.csdn.net/feiyangqingyun/article/details/97565652
4. Experience address: https://pan.baidu.com/s/1d7TH_GEYl5nOecuNlWJJ7g Extraction code: 01jf File name: bin_video_demo/bin_linux_video.

4. Related codes

void AbstractVideoThread::checkDeviceUrl(const QString &url, QString &deviceName, QString &resolution, int &frameRate)
{
    
    
    int offsetX, offsetY;
    AbstractVideoThread::checkDeviceUrl(url, deviceName, resolution, frameRate, offsetX, offsetY);
}

void AbstractVideoThread::checkDeviceUrl(const QString &url, QString &deviceName, QString &resolution, int &frameRate, int &offsetX, int &offsetY)
{
    
    
    //无论是否带分隔符第一个约定是设备名称
    QStringList list = url.split("|");
    deviceName = list.at(0);

    //带分隔符说明还指定了分辨率或帧率
    if (url.contains("|")) {
    
    
        QStringList sizes;
        QString size = list.at(1);
        if (size.contains("*")) {
    
    
            sizes = size.split("*");
        } else if (size.contains("x")) {
    
    
            sizes = size.split("x");
        }

        if (sizes.count() == 2) {
    
    
            int width = sizes.at(0).toInt();
            int height = sizes.at(1).toInt();
            resolution = QString("%1x%2").arg(width).arg(height);
        } else {
    
    
            resolution = "0x0";
        }

        //第三个参数是帧率
        if (list.count() >= 3) {
    
    
            frameRate = list.at(2).toInt();
        }

        //第四第五个参数是XY坐标偏移值
        if (list.count() >= 5) {
    
    
            offsetX = list.at(3).toInt();
            offsetY = list.at(4).toInt();
        }
    }

    //设置个默认的分辨率(几乎所有本地设备都支持这个分辨率)
    if (resolution == "0x0" && (url.startsWith("video=") || url.startsWith("/dev"))) {
    
    
        resolution = "640x480";
    }
}

void VideoHelper::getDesktopPara(const VideoCore &videoCore, QString &videoUrl, QString &bufferSize, int &frameRate, int &offsetX, int &offsetY)
{
    
    
    //ffmpeg -f gdigrab -s 800x600 -r 30 -i desktop out.mp4
    //ffmpeg -f x11grab -s 800x600 -r 30 -i :0.0+0,0 out.mp4
    //ffmpeg -f avfoundation -s 800x600 -r 30 -i 0:0 out.mp4

    QString url = videoUrl;
    url.replace("desktop=", "");
    url.replace("screen://", "");
    url.replace("avdevice:gdigrab:", "");
    AbstractVideoThread::checkDeviceUrl(url, videoUrl, bufferSize, frameRate, offsetX, offsetY);

    //不同内核对应地址不一样
    if (videoCore == VideoCore_FFmpeg) {
    
    
#if defined(Q_OS_WIN)
        url = "desktop";
#elif defined(Q_OS_LINUX)
        url = ":0.0+0,0";
#elif defined(Q_OS_MAC)
        url = "0:0";
#endif
    } else if (videoCore == VideoCore_Vlc) {
    
    
        url = "screen://";
    } else if (videoCore == VideoCore_Qtav) {
    
    
        url = "avdevice:gdigrab:desktop";
    } else if (videoCore == VideoCore_EasyPlayer) {
    
    
        url = "gdigrab://desktop";
    }

    videoUrl = url;
}

5. Features

5.1 Basic functions

1. Support various audio and video file formats, such as mp3, wav, mp4, asf, rm, rmvb, mkv, etc.
2. Support local camera equipment, you can specify the resolution and frame rate.
3. Support various video streaming formats, such as rtp, rtsp, rtmp, http, etc.
4. Local audio and video files and network audio and video files, automatically identify file length, playback progress, volume, mute status, etc.
5. The file can specify the playback position, adjust the volume, set the mute status, etc.
6. Supports double-speed playback of files, and can choose 0.5 times, 1.0 times, 2.5 times, 5.0 times and other speeds, which is equivalent to slow playback and fast playback.
7. Support start playback, stop playback, pause playback, continue playback.
8. Supports snapping screenshots, you can specify the file path, and you can choose whether to automatically display the preview after the snapping is completed.
9. Support video storage, manually start and stop recording, some kernels support continuing recording after pausing recording, and skip the part that does not need to be recorded.
10. Support mechanisms such as non-perceptual switching loop playback, automatic reconnection, etc.
11. Provides signals such as successful playback, playback completion, received decoded pictures, received captured pictures, video size changes, and recording status changes.
12. Multi-thread processing, one decoding thread, no stuck on the main interface.

5.2 Features

1. Support multiple decoding kernels at the same time, including qmedia kernel (Qt4/Qt5/Qt6), ffmpeg kernel (ffmpeg2/ffmpeg3/ffmpeg4/ffmpeg5), vlc kernel (vlc2/vlc3), mpv kernel (mpv1/mp2), Hikvision SDK, easyplayer kernel etc.
2. Very complete multiple base class design, adding a new decoding core only needs to implement a very small amount of code, and the whole set of mechanisms can be applied.
3. At the same time, it supports a variety of screen display strategies, automatic adjustment (the original resolution is smaller than the size of the display control, it will be displayed according to the original resolution, otherwise it will be scaled proportionally), proportional scaling (forever proportional scaling), stretching and filling (forever stretching and filling ). Three screen display strategies are supported in all kernels and in all video display modes.
4. At the same time, it supports a variety of video display modes, handle mode (pass in the control handle to the other party for drawing control), draw mode (call back to get the data and convert it to QImage and draw it with QPainter), GPU mode (call back to get the data and convert it to yuv for use QOpenglWidget draws).
5. Support multiple hardware acceleration types, ffmpeg can choose dxva2, d3d11va, etc., mpv can choose auto, dxva2, d3d11va, vlc can choose any, dxva2, d3d11va. Different system environments have different types of options, such as vaapi and vdpau for linux systems, and videotoolbox for macos systems.
6. The decoding thread is separated from the display window, and any decoding core can be specified to be mounted to any display window and switched dynamically.
7. It supports shared decoding thread, which is enabled by default and automatically processed. When the same video address is recognized, a decoding thread is shared, which can greatly save network traffic and the push pressure of the other party's device in the network video environment. Top domestic video manufacturers all adopt this strategy. In this way, as long as one video stream is pulled, it can be shared to dozens or hundreds of channels for display.
8. Automatically identify the video rotation angle and draw it. For example, the video shot on a mobile phone is generally rotated by 90 degrees. It must be automatically rotated during playback, otherwise the default is upside down.
9. Automatically recognizes resolution changes during video streaming and automatically adjusts the size on the video controls. For example, the camera can dynamically configure the resolution during use, and when the resolution changes, the corresponding video controls also respond synchronously.
10. Audio and video files are automatically switched and played in a loop without perception, and there will be no visible switching traces such as black screens during switching.
11. The video control also supports any decoding core, any screen display strategy, and any video display mode.
12. The video control floating bar supports three modes of handle, drawing, and GPU at the same time, and the non-absolute coordinates can be moved around.
13. The local camera device supports playing by specifying the device name, resolution, and frame rate.
14. Recording files also support open video files, local cameras, network video streams, etc.
15. Respond to opening and closing instantly, whether it is opening a video or network stream that does not exist, detecting the existence of a device, waiting for a timeout in reading, and immediately interrupting the previous operation and responding when the close command is received.
16. Support to open various picture files, and support local audio and video files to drag and play.
17. The video control floating bar comes with functions such as start and stop recording switching, sound mute switching, snapping screenshots, and closing video.
18. The audio component supports sound waveform value data analysis, and can draw waveform curves and columnar sound bars based on the value, and provides sound amplitude signals by default.
19. The extremely detailed print information prompts in each component, especially the error message prompts, and the unified print format of the package. It is extremely convenient and useful to test the complex equipment environment on site, which is equivalent to accurately locating which channel and which step are wrong.
20. The code framework and structure are optimized to the best, the performance is powerful, and iteratively updated and upgraded continuously.
21. The source code supports Qt4, Qt5, Qt6, compatible with all versions.

5.3 Video Controls

1. Any number of osd label information can be added dynamically. Label information includes name, whether it is visible, font size, text text, text color, label picture, label coordinates, label format (text, date, time, date time, picture), label position (upper left, lower left, upper right, lower right, centered, custom coordinates).
2. Any number of graphics information can be dynamically added, which is very useful, for example, the graphics area information analyzed by the artificial intelligence algorithm can be directly sent to the video control. The graphic information supports any shape, which can be directly drawn on the original picture with absolute coordinates.
3. Graphic information includes name, border size, border color, background color, rectangular area, path collection, point coordinate collection, etc.
4. One or more of the three types of areas can be specified for each graphic information, and all specified areas will be drawn.
5. Built-in floating bar control, the floating bar position supports top, bottom, left, right.
6. The parameters of the floating bar control include margin, spacing, background transparency, background color, text color, pressed color, position, button icon code collection, button name identification collection, and button prompt information collection.
7. A row of tool buttons in the floating bar control can be customized. Through the structure parameter setting, the icon can choose a graphic font or a custom picture.
8. The floating bar button internally realizes functions such as video switching, snapping screenshots, mute switching, and turning off video, and you can also add your own corresponding functions in the source code.
9. The floating bar button corresponds to the button that has realized the function, and there is a corresponding icon switching process. For example, after the recording button is pressed, it will switch to the icon that is being recorded. After the sound button is switched, it will become a mute icon, and then switch again to restore.
10. After the button of the floating bar is clicked, it will be sent as a signal with the unique identification of the name, and it can be associated with the response processing by itself.
11. Prompt information can be displayed in the blank area of ​​the floating bar. By default, the current video resolution is displayed, and information such as frame rate and code stream size can be added.
12. Video control parameters include border size, border color, focus color, background color (transparent by default), text color (default global text color), fill color (blank space outside the video is filled with black), background text, background image (if set Pictures are preferred), whether to copy pictures, scaling display mode (automatic adjustment, proportional scaling, stretching and filling), video display mode (handle, drawing, GPU), enable floating bar, floating bar size (horizontal is height, vertical is the width), the position of the floating bar (top, bottom, left, right).