Qt/C++ audio and video development 49-multi-level continuous storage and streaming design (save to multiple files/push to multiple platforms at the same time)

I. Introduction

Recently, I encountered a user who needed multi-cascade saving and streaming. To implement this function in the ffmpegsave multi-threaded saving class, the simpler the better, that is, the automatic dump function can be turned on while pushing the stream, and the video can be saved to a local video file while pushing the stream. The original plan was two new ffmpegsave classes, one dedicated to stream streaming and one dedicated to video recording. It is definitely possible, but the performance is low, because the packet data packet or the encoded packet, whether it is the source 264/265 data packet or the encoded packet, has already been obtained. It only needs one storage action. Why do you need another save class to do this action again? This flag marks whether the packet needs to be resent out. If it is enabled, it will send out the legal and complete packet signal before streaming. The other save class only needs to call the writepacket interface function and write it directly to the file. This is the most efficient processing. If the source is rtsp video stream (usually 264/265/aac), then the video storage at the same time as streaming, takes up almost 0% of the CPU, and only 0.1% fluctuates when two channels of 1080P are opened. The performance is not too high, because there is no need for the encoding process, which is equivalent to writing files or streaming data all the time. The main performance bottleneck lies in the read and write speed of the network and hard disk.

2. Rendering

3. Experience address

1. Domestic site: https://gitee.com/feiyangqingyun
2. International site: https://github.com/feiyangqingyun
3. Personal works: https://blog.csdn.net/feiyangqingyun/article/details/97565652
4. Experience address: https://pan.baidu.com/s/1d7TH_GEYl5nOecuNlWJJ7g Extraction code: 01jf File name: bin_video_demo.

4. Features

4.1 Basic functions

1. Support various audio and video file formats, such as mp3, wav, mp4, asf, rm, rmvb, mkv, etc.
2. Support local camera equipment, you can specify the resolution and frame rate.
3. Support various video streaming formats, such as rtp, rtsp, rtmp, http, etc.
4. Local audio and video files and network audio and video files, automatically identify file length, playback progress, volume, mute status, etc.
5. The file can specify the playback position, adjust the volume, set the mute status, etc.
6. Supports double-speed playback of files, and can choose 0.5 times, 1.0 times, 2.5 times, 5.0 times and other speeds, which is equivalent to slow playback and fast playback.
7. Support start playback, stop playback, pause playback, continue playback.
8. Supports snapping screenshots, you can specify the file path, and you can choose whether to automatically display the preview after the snapping is completed.
9. Support video storage, manually start and stop recording, some kernels support continuing recording after pausing recording, and skip the part that does not need to be recorded.
10. Support mechanisms such as non-perceptual switching loop playback, automatic reconnection, etc.
11. Provides signals such as successful playback, playback completion, received decoded pictures, received captured pictures, video size changes, and recording status changes.
12. Multi-thread processing, one decoding thread, no stuck on the main interface.

4.2 Features

1. At the same time, it supports multiple decoding kernels, including qmedia kernel (Qt4/Qt5/Qt6), ffmpeg kernel (ffmpeg2/ffmpeg3/ffmpeg4/ffmpeg5), vlc kernel (vlc2/vlc3), mpv kernel (mpv1/mp2), monitoring manufacturer SDK kernel, etc.
2. Very complete multiple base class design, adding a new decoding core only needs to implement a very small amount of code, and the whole set of mechanisms can be applied.
3. At the same time, it supports a variety of screen display strategies, automatic adjustment (the original resolution is smaller than the size of the display control, it will be displayed according to the original resolution size, otherwise it will be scaled proportionally), proportional scaling (proportional scaling forever), stretching and filling (stretching and filling forever). Three screen display strategies are supported in all kernels and in all video display modes.
4. At the same time, it supports a variety of video display modes, handle mode (pass in the control handle to the other party for drawing control), draw mode (call back to get the data and convert it to QImage and use QPainter to draw), GPU mode (call back to get the data and then convert it to yuv and use QOpenglWidget to draw).
5. Support multiple hardware acceleration types, ffmpeg can choose dxva2, d3d11va, etc., mpv can choose auto, dxva2, d3d11va, vlc can choose any, dxva2, d3d11va. Different system environments have different types of options, such as vaapi and vdpau for linux systems, and videotoolbox for macos systems.
6. The decoding thread is separated from the display window, and any decoding core can be specified to be mounted to any display window and switched dynamically.
7. Support shared decoding thread, which is enabled by default and automatically processed. When the same video address is recognized, a decoding thread is shared, which can greatly save network traffic and the push pressure of the other party's device in the network video environment. Top domestic video manufacturers all adopt this strategy. In this way, as long as one video stream is pulled, it can be shared to dozens or hundreds of channels for display.
8. Automatically identify the video rotation angle and draw it. For example, the video shot on a mobile phone is generally rotated by 90 degrees. It must be automatically rotated during playback, otherwise the default is upside down.
9. Automatically recognizes resolution changes during video streaming and automatically adjusts the size on the video controls. For example, the camera can dynamically configure the resolution during use, and when the resolution changes, the corresponding video controls also respond synchronously.
10. Audio and video files are automatically switched and played in a loop without perception, and there will be no visible switching traces such as black screens during switching.
11. The video control also supports any decoding core, any screen display strategy, and any video display mode.
12. The video control floating bar supports three modes of handle, drawing, and GPU at the same time, and the non-absolute coordinates can be moved around.
13. The local camera device supports playing by specifying the device name, resolution, and frame rate.
14. Recording files also support open video files, local cameras, network video streams, etc.
15. Respond to opening and closing instantly, whether it is opening a video or network stream that does not exist, detecting the existence of a device, waiting for a timeout in reading, and immediately interrupting the previous operation and responding when the close command is received.
16. Support to open various picture files, and support local audio and video files to drag and play.
17. The video control floating bar comes with functions such as start and stop recording switching, sound mute switching, snapping screenshots, and closing video.
18. The audio component supports sound waveform value data analysis, and can draw waveform curves and columnar sound bars based on the value, and provides sound amplitude signals by default.
19. The code framework and structure are optimized to the extreme, the performance is sturdy, and iteratively updated and upgraded continuously.
20. The source code supports Qt4, Qt5, Qt6, compatible with all versions.

4.3 Video Controls

1. Any number of osd label information can be dynamically added. Label information includes name, visibility, font size, text, text color, label picture, label coordinates, label format (text, date, time, date time, picture), label position (upper left corner, lower left corner, upper right corner, lower right corner, center, custom coordinates).
2. Any number of graphics information can be dynamically added, which is very useful, for example, the graphics area information analyzed by the artificial intelligence algorithm can be directly sent to the video control. The graphic information supports any shape, which can be directly drawn on the original picture with absolute coordinates.
3. Graphic information includes name, border size, border color, background color, rectangular area, path collection, point coordinate collection, etc.
4. One or more of the three types of areas can be specified for each graphic information, and all specified areas will be drawn.
5. Built-in floating bar control, the floating bar position supports top, bottom, left, right.
6. The parameters of the floating bar control include margin, spacing, background transparency, background color, text color, pressed color, position, button icon code collection, button name identification collection, and button prompt information collection.
7. A row of tool buttons in the floating bar control can be customized. Through the structure parameter setting, the icon can choose a graphic font or a custom picture.
8. The floating bar button internally realizes functions such as video switching, snapping screenshots, mute switching, and turning off video, and you can also add your own corresponding functions in the source code.
9. The floating bar button corresponds to the button that has realized the function, and there is a corresponding icon switching process. For example, after the recording button is pressed, it will switch to the icon that is being recorded. After the sound button is switched, it will become a mute icon, and then switch again to restore.
10. After the button of the floating bar is clicked, it will be sent as a signal with the unique identification of the name, and it can be associated with the response processing by itself.
11. Prompt information can be displayed in the blank area of ​​the floating bar. By default, the current video resolution is displayed, and information such as frame rate and code stream size can be added.
12. Video control parameters include border size, border color, focus color, background color (transparent by default), text color (default global text color), fill color (fill the blank space outside the video with black), background text, background image (if the image is set to take priority), whether to copy the image, zoom display mode (automatic adjustment, proportional scaling, stretch to fill), video display mode (handle, drawing, GPU), enable floating bar, floating bar size (horizontal is height, vertical is width), floating bar position (top, bottom, left, right).

4.4 kernel ffmpeg

1. Supports various audio and video files, local camera equipment, and various video stream network streams.
2. Support start playback, pause playback, continue playback, stop playback, set playback progress, double speed playback.
3. You can set the volume, mute switching, snap pictures, video storage.
4. Automatically extract album information such as title, artist, album, album cover, and automatically display the album cover.
5. Perfect support for audio and video synchronization and double-speed playback.
6. The decoding strategy supports speed priority, quality priority, equalization processing, and the fastest speed.
7. Support mobile phone video rotation angle display. For example, the video shot by a general mobile phone is rotated 90 degrees. When decoding and displaying, it needs to be rotated 90 degrees again to be correct.
8. Automatically convert yuv420 format, for example, the local camera is in yuyv422 format, some video files are in xx format, and the non-yuv420 format will be converted uniformly, and then processed.
9. Support hard decoding dxva2, d3d11va, etc., with extremely high performance, especially for large resolution such as 4K video.
10. The video response is extremely low and the delay is about 0.2s, and the fast response is about 0.5s to open the video stream, which is specially optimized.
11. Combination of hardware decoding and GPU drawing, extremely low CPU usage, better than clients such as Hikvision Dahua.
12. Supports various audio formats in video streams, including AAC, PCM, G.726, G.711A, G.711Mu, G.711ulaw, G.711alaw, MP2L2, etc. It is recommended to choose AAC for compatibility and cross-platform.
13. Video storage supports yuv, h264, mp4 formats, audio storage supports pcm, wav, aac formats. Default video mp4 format, audio aac format.
14. It supports separate storage of audio and video files, and also supports merging into one mp4 file. The default strategy is that no matter what audio and video file format is stored, it will eventually be converted into mp4 and aac formats, and then merged into an mp4 file with both audio and video.
15. Support local camera real-time video display with audio input and output, audio and video recording merged into one mp4 file.
16. Support H265 encoding (now more and more surveillance cameras are in H265 video stream format) to generate video files, and automatically identify and switch encoding formats internally.
17. Automatically recognize the dynamic resolution change of the video stream and reopen the video stream.
18. Pure qt+ffmpeg decoding, non-sdl and other third-party rendering and playback dependencies, gpu rendering uses qopenglwidget, audio playback uses qaudiooutput.
19. At the same time, it supports ffmpeg2, ffmpeg3, ffmpeg4, and ffmpeg5 versions, all of which are compatible. If you need to support xp, you need to choose ffmpeg3 and below.

5. Related codes

void FFmpegPushClient::receivePlayStart(int time)
{
    
    
    //打开后才能启动录像
    ffmpegThread->recordStart(pushUrl);

    //推流以外还单独存储
    if (!fileName.isEmpty() && !ffmpegSave) {
    
    
        //源头保存没成功就不用继续
        FFmpegSave *saveFile = ffmpegThread->getSaveFile();
        if (!saveFile->getIsOk()) {
    
    
            return;
        }

        ffmpegSave = new FFmpegSave(this);
        //重新编码过的则取视频保存类的对象
        AVStream *videoStreamIn = saveFile->getEncodeVideo() ? saveFile->getVideoStream() : ffmpegThread->getVideoStream();
        AVStream *audioStreamIn = saveFile->getEncodeAudio() ? saveFile->getAudioStream() : ffmpegThread->getAudioStream();
        ffmpegSave->setPara(ffmpegThread->getVideoType(), SaveVideoType_Mp4, videoStreamIn, audioStreamIn);
        ffmpegSave->open(fileName);
    }
}

void FFmpegPushClient::receivePacket(AVPacket *packet)
{
    
    
    if (ffmpegSave) {
    
    
        ffmpegSave->writePacket2(packet);
    }

    FFmpegHelper::freePacket(packet);
}

void FFmpegPushClient::recorderStateChanged(const RecorderState &state, const QString &file)
{
    
    
    int width = 0;
    int height = 0;
    int videoStatus = 0;
    int audioStatus = 0;
    if (ffmpegThread) {
    
    
        width = ffmpegThread->getVideoWidth();
        height = ffmpegThread->getVideoHeight();
        FFmpegSave *saveFile = ffmpegThread->getSaveFile();
        if (saveFile->getIsOk()) {
    
    
            if (saveFile->getVideoIndexIn() >= 0) {
    
    
                if (saveFile->getVideoIndexOut() >= 0) {
    
    
                    videoStatus = (saveFile->getEncodeVideo() ? 3 : 2);
                } else {
    
    
                    videoStatus = 1;
                }
            }
            if (saveFile->getAudioIndexIn() >= 0) {
    
    
                if (saveFile->getAudioIndexOut() >= 0) {
    
    
                    audioStatus = (saveFile->getEncodeAudio() ? 3 : 2);
                } else {
    
    
                    audioStatus = 1;
                }
            }
        }
    }

    //只有处于录制中才表示正常推流开始
    bool start = (state == RecorderState_Recording);
    emit pushStart(videoUrl, width, height, videoStatus, audioStatus, start);
}

void FFmpegPushClient::start()
{
    
    
    if (ffmpegThread || videoUrl.isEmpty() || pushUrl.isEmpty()) {
    
    
        return;
    }

    //实例化视频采集线程
    ffmpegThread = new FFmpegThread;
    //关联播放开始信号用来启动推流
    connect(ffmpegThread, SIGNAL(receivePlayStart(int)), this, SLOT(receivePlayStart(int)));
    //关联录制信号变化用来判断是否推流成功
    connect(ffmpegThread, SIGNAL(recorderStateChanged(RecorderState, QString)), this, SLOT(recorderStateChanged(RecorderState, QString)));
    //设置播放地址
    ffmpegThread->setVideoUrl(videoUrl);
    //设置解码内核
    ffmpegThread->setVideoCore(VideoCore_FFmpeg);
    //设置视频模式
    ffmpegThread->setVideoMode(VideoMode_Opengl);
    //设置硬解码(和推流无关/只是为了加速显示/推流只和硬编码有关)
    //ffmpegThread->setHardware("dxva2");
    //设置解码策略(推流的地址再拉流建议开启最快速度)
    //ffmpegThread->setDecodeType(DecodeType_Fastest);
    //设置读取超时时间超时后会自动重连
    ffmpegThread->setReadTimeout(5 * 1000);
    //设置连接超时时间(0表示一直连)
    ffmpegThread->setConnectTimeout(0);
    //设置重复播放相当于循环推流
    ffmpegThread->setPlayRepeat(true);
    //设置默认不播放音频(界面上切换到哪一路就开启)
    ffmpegThread->setPlayAudio(false);
    //设置默认不预览视频(界面上切换到哪一路就开启)
    ffmpegThread->setPushPreview(false);

    //设置保存视频类将数据包信号发出来用于保存文件
    FFmpegSave *saveFile = ffmpegThread->getSaveFile();
    saveFile->setSendPacket(!fileName.isEmpty(), false);
    connect(saveFile, SIGNAL(receivePacket(AVPacket *)), this, SLOT(receivePacket(AVPacket *)));
    connect(saveFile, SIGNAL(receiveSaveStart()), this, SLOT(receiveSaveStart()));
    connect(saveFile, SIGNAL(receiveSaveFinsh()), this, SLOT(receiveSaveFinsh()));
    connect(saveFile, SIGNAL(receiveSaveError(int)), this, SLOT(receiveSaveError(int)));

    //演示添加OSD(可以多个)后推流
    //this->initOsd();

    //如果是本地设备或者桌面录屏要取出其他参数
    VideoHelper::initVideoPara(ffmpegThread, videoUrl, encodeVideoScale);

    //设置编码策略/视频压缩比率/视频缩放比例
    ffmpegThread->setEncodeType((EncodeType)encodeType);
    ffmpegThread->setEncodeVideoRatio(encodeVideoRatio);
    ffmpegThread->setEncodeVideoScale(encodeVideoScale);

    //启动播放
    ffmpegThread->play();
}