Design of ffmpeg video pause recording

1 Introduction

Basically, the recording video interfaces provided by various players only have two interfaces: start recording and end recording. Of course, these two interfaces are generally used the most, but in the actual use process, there is another possibility that the recording needs to be paused midway. After pausing, continue recording again, skip the unnecessary video in the middle part of the video, and the recorded video file must be continuous and playable normally. Both vlc and mpv only provide interfaces to start recording and stop recording. Since ffmpeg decodes by itself, the recording is completely controlled by itself. When storing, the data of each frame must be written into pts and dts, and the time base is recalculated each time to ensure The time is correct, otherwise non-continuous frames will skip after being saved.

After opening the function of video pause recording, there is another application scenario that can solve the problem, that is, videos of multiple channels are stored in the same video file at different time intervals, which is similar to storing the videos displayed in the polling process one by one. For the same video file, of course, the resolution must be consistent. Inconsistency may cause storage problems. This kind of scene is quite common. The entire process of the inquiry is recorded, and the video can be played back later. In the currently packaged components, in addition to turning on and off the recording, you only need to pass in the video frame halfway, and it will automatically calculate and convert it into the correct pts/dts and store it in the MP4 file.

2. Rendering

3. Experience address

Domestic site: gitee.com/feiyangqing…
International site: github.com/feiyangqing…
Personal works: blog.csdn.net/feiyangqing…
Experience address: pan.baidu.com/s/1d7TH_GEY ... Extraction code: 01jf File name: bin_video_demo/bin_linux_video.

4. Related codes

void FFmpegThread::recordStart(const QString &fileName)
{
#ifdef videosave
    AbstractVideoThread::recordStart(fileName);
    if ((saveVideoType > 1) && !onlyAudio) {
        this->setFileName(fileName);
        //处于暂停阶段则切换暂停标志位(暂停后再次恢复说明又重新开始录制)
        if (saveFile->getIsPause()) {
            isRecord = true;
            saveFile->pause();
            emit recorderStateChanged(RecorderState_Recording, fileName);
        } else {
            saveFile->setPara(saveVideoType, videoWidth, videoHeight, frameRate, formatCtx->streams[videoIndex], (videoType == VideoType_Camera));
            saveFile->open(fileName);
            if (saveFile->getIsOk()) {
                isRecord = true;
                emit recorderStateChanged(RecorderState_Recording, fileName);
            }
        }
    }
#endif
}

void FFmpegThread::recordPause()
{
#ifdef videosave
    AbstractVideoThread::recordPause();
    if ((saveVideoType > 1) && !onlyAudio) {
        if (saveFile->getIsOk()) {
            isRecord = false;
            saveFile->pause();
            emit recorderStateChanged(RecorderState_Paused, fileName);
        }
    }
#endif
}

void FFmpegThread::recordStop()
{
#ifdef videosave
    AbstractVideoThread::recordStop();
    if ((saveVideoType > 1) && !onlyAudio) {
        if (saveFile->getIsOk()) {
            isRecord = false;
            saveFile->stop();
            //执行过转换合并的不用再发信号
            if (!saveFile->isConvertMerge) {
                emit recorderStateChanged(RecorderState_Stopped, fileName);
            }
        }
    }
#endif
}

void FFmpegSave::save()
{
    //从队列中取出数据处理
    //qDebug() << TIMEMS << videoFrames.count() << videoPackets.count();
    if (videoFrames.count() > 0) {
        mutex.lock();
        AVFrame *frame = videoFrames.takeFirst();
        mutex.unlock();
        FFmpegHelper::encode(this, videoCodecCtx, videoPacket, frame, true);
        FFmpegHelper::freeFrame(frame);
    }

    if (videoPackets.count() > 0) {
        mutex.lock();
        AVPacket *packet = videoPackets.takeFirst();
        mutex.unlock();
        this->writePacket(packet);
        FFmpegHelper::freePacket(packet);
    }
}

void FFmpegSave::close()
{
    //写入过开始符才能写入文件结束符(没有这个判断会报错)
    if (packetCount > 0 && saveVideoType == SaveVideoType_Mp4) {
        av_write_trailer(formatCtx);
    }

    //清空队列中的数据
    foreach (AVFrame *frame, videoFrames) {
        FFmpegHelper::freeFrame(frame);
    }

    foreach (AVPacket *packet, videoPackets) {
        FFmpegHelper::freePacket(packet);
    }

    packetCount = 0;
    videoFrames.clear();
    videoPackets.clear();

    //释放临时数据包
    if (videoPacket) {
        FFmpegHelper::freePacket(videoPacket);
        videoPacket = NULL;
    }

    //关闭编码器上下文并释放对象
    if (videoCodecCtx) {
        avcodec_free_context(&videoCodecCtx);
        videoCodec = NULL;
        videoCodecCtx = NULL;
    }

    //关闭文件流并释放对象
    if (formatCtx) {
        avio_close(formatCtx->pb);
        avformat_free_context(formatCtx);
        formatCtx = NULL;
        videoStreamOut = NULL;
        videoStreamIn = NULL;
    }

    //执行转换合并音频文件到一个文件
    isConvertMerge = false;
    if (convertMerge) {
        if (saveVideoType == SaveVideoType_H264) {
            isConvertMerge = FFmpegRun::aacAndH264ToMp4(fileName);
        } else if (saveVideoType == SaveVideoType_Mp4) {
            isConvertMerge = FFmpegRun::aacAndMp4ToMp4(fileName);
        }
    }
}

void FFmpegSave::setPara(const SaveVideoType &saveVideoType, int videoWidth, int videoHeight, int frameRate, AVStream *videoStreamIn, bool camera)
{
    this->saveVideoType = saveVideoType;
    this->videoWidth = videoWidth;
    this->videoHeight = videoHeight;
    if (camera) {
        this->frameRate = frameRate > 15 ? 16.621 : frameRate;
    } else {
        this->frameRate = frameRate > 25 ? 25 : frameRate;
    }
    this->videoStreamIn = videoStreamIn;
}

void FFmpegSave::writeVideo(AVFrame *frame)
{
    //没打开或者暂停阶段不处理
    if (!isOk || isPause) {
        return;
    }

    //可以直接写入到文件也可以排队处理
    if (directSave) {
        FFmpegHelper::encode(this, videoCodecCtx, videoPacket, frame, true);
    } else {
        mutex.lock();
        videoFrames << av_frame_clone(frame);
        mutex.unlock();
    }
}

void FFmpegSave::writeVideo(AVPacket *packet)
{
    //没打开或者暂停阶段不处理
    if (!isOk || isPause) {
        return;
    }

    //可以直接写入到文件也可以排队处理
    if (directSave) {
        this->writePacket(packet);
    } else {
        mutex.lock();
        videoPackets << FFmpegHelper::creatPacket(packet);
        mutex.unlock();
    }
}

5. Features

5.1 Basic functions

Support various audio and video file formats, such as mp3, wav, mp4, asf, rm, rmvb, mkv, etc.
Support local camera equipment, you can specify the resolution and frame rate.
Support various video streaming formats, such as rtp, rtsp, rtmp, http, etc.
Local audio and video files and network audio and video files, automatically identify file length, playback progress, volume, mute status, etc.
The file can specify the playback position, adjust the volume, set the mute status, etc.
Supports double-speed playback of files, and can choose 0.5 times, 1.0 times, 2.5 times, 5.0 times and other speeds, which is equivalent to slow playback and fast playback.
Support start playback, stop playback, pause playback, continue playback.
Supports snapping screenshots, you can specify the file path, and you can choose whether to automatically display the preview after the snapping is completed.
Support video storage, manually start and stop recording, some kernels support continuing recording after pausing recording, and skip the part that does not need to be recorded.
Support mechanisms such as non-perceptual switching loop playback, automatic reconnection, etc.
Provides signals such as successful playback, playback completion, received decoded pictures, received captured pictures, video size changes, and recording status changes.
Multi-thread processing, one decoding thread, no stuck on the main interface.

[Learning address]: FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media advanced development

[Article Benefits]: Receive more audio and video learning packages, Dachang interview questions, technical videos and learning roadmaps for free. The materials include (C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) Click 1079654574 to join the group to receive it~

5.2 Features

Support multiple decoding kernels at the same time, including qmedia kernel (Qt4/Qt5/Qt6), ffmpeg kernel (ffmpeg2/ffmpeg3/ffmpeg4/ffmpeg5), vlc kernel (vlc2/vlc3), mpv kernel (mpv1/mp2), Hikvision SDK, easyplayer kernel etc.
Very complete multiple base class design, adding a new decoding core only needs to implement a very small amount of code, and the whole set of mechanisms can be applied.
At the same time, it supports a variety of screen display strategies, automatic adjustment (the original resolution is smaller than the size of the display control, it will be displayed according to the original resolution, otherwise it will be scaled proportionally), proportional scaling (forever proportional scaling), stretching and filling (forever stretching and filling ). Three screen display strategies are supported in all kernels and in all video display modes.
At the same time, it supports a variety of video display modes, handle mode (pass in the control handle to the other party for drawing control), draw mode (call back to get the data and convert it to QImage and draw it with QPainter), GPU mode (call back to get the data and convert it to yuv for use QOpenglWidget draws).
Support multiple hardware acceleration types, ffmpeg can choose dxva2, d3d11va, etc., mpv can choose auto, dxva2, d3d11va, vlc can choose any, dxva2, d3d11va. Different system environments have different types of options, such as vaapi and vdpau for linux systems, and videotoolbox for macos systems.
The decoding thread is separated from the display window, and any decoding core can be specified to be mounted to any display window and switched dynamically.
Support shared decoding thread, which is enabled by default and automatically processed. When the same video address is recognized, a decoding thread is shared, which can greatly save network traffic and the push pressure of the other party's device in the network video environment. Top domestic video manufacturers all adopt this strategy. In this way, as long as one video stream is pulled, it can be shared to dozens or hundreds of channels for display.
Automatically identify the video rotation angle and draw it. For example, the video shot on a mobile phone is generally rotated by 90 degrees. It must be automatically rotated during playback, otherwise the default is upside down.
Automatically recognizes resolution changes during video streaming and automatically adjusts the size on the video controls. For example, the camera can dynamically configure the resolution during use, and when the resolution changes, the corresponding video controls also respond synchronously.
Audio and video files are automatically switched and played in a loop without perception, and there will be no visible switching traces such as black screens during switching.
The video control also supports any decoding core, any screen display strategy, and any video display mode.
The video control floating bar supports three modes of handle, drawing, and GPU at the same time, and the non-absolute coordinates can be moved around.
The local camera device supports playing by specifying the device name, resolution, and frame rate.
Recording files also support open video files, local cameras, network video streams, etc.
Respond to opening and closing instantly, whether it is opening a video or network stream that does not exist, detecting the existence of a device, waiting for a timeout in reading, and immediately interrupting the previous operation and responding when the close command is received.
Support to open various picture files, and support local audio and video files to drag and play.
The video control floating bar comes with functions such as start and stop recording switching, sound mute switching, snapping screenshots, and closing video.
The audio component supports sound waveform value data analysis, and can draw waveform curves and columnar sound bars based on the value, and provides sound amplitude signals by default.
The extremely detailed print information prompts in each component, especially the error message prompts, and the unified print format of the package. It is extremely convenient and useful to test the complex equipment environment on site, which is equivalent to accurately locating which channel and which step are wrong.
The code framework and structure are optimized to the best, the performance is powerful, and iteratively updated and upgraded continuously.
The source code supports Qt4, Qt5, Qt6, compatible with all versions.

5.3 Video Controls

Any number of osd label information can be added dynamically. Label information includes name, whether it is visible, font size, text text, text color, label picture, label coordinates, label format (text, date, time, date time, picture), label position (upper left, lower left, upper right, lower right, centered, custom coordinates).
Any number of graphics information can be dynamically added, which is very useful, for example, the graphics area information analyzed by the artificial intelligence algorithm can be directly sent to the video control. The graphic information supports any shape, which can be directly drawn on the original picture with absolute coordinates.
Graphic information includes name, border size, border color, background color, rectangular area, path collection, point coordinate collection, etc.
One or more of the three types of areas can be specified for each graphic information, and all specified areas will be drawn.
Built-in floating bar control, the floating bar position supports top, bottom, left, right.
The parameters of the floating bar control include margin, spacing, background transparency, background color, text color, pressed color, position, button icon code collection, button name identification collection, and button prompt information collection.
A row of tool buttons in the floating bar control can be customized. Through the structure parameter setting, the icon can choose a graphic font or a custom picture.
The floating bar button internally realizes functions such as video switching, snapping screenshots, mute switching, and turning off video, and you can also add your own corresponding functions in the source code.
The floating bar button corresponds to the button that has realized the function, and there is a corresponding icon switching process. For example, after the recording button is pressed, it will switch to the icon that is being recorded. After the sound button is switched, it will become a mute icon, and then switch again to restore.
After the button of the floating bar is clicked, it will be sent as a signal with the unique identification of the name, and it can be associated with the response processing by itself.
Prompt information can be displayed in the blank area of the floating bar. By default, the current video resolution is displayed, and information such as frame rate and code stream size can be added.
Video control parameters include border size, border color, focus color, background color (transparent by default), text color (default global text color), fill color (blank space outside the video is filled with black), background text, background image (if set Pictures are preferred), whether to copy pictures, scaling display mode (automatic adjustment, proportional scaling, stretching and filling), video display mode (handle, drawing, GPU), enable floating bar, floating bar size (horizontal is height, vertical is the width), the position of the floating bar (top, bottom, left, right).

Original link: Qt audio and video development 38-ffmpeg video pause recording design - Nuggets