FFmpeg development notes (6): ffmpeg decodes the video and uses SDL to synchronize time display and playback

If the article is an original article, it cannot be reprinted without permission.
Original blogger’s blog address: https://blog.csdn.net/qq21497936
Original blogger’s blog navigation: https://blog.csdn.net/qq21497936/article/details /102478062
The blog address of this article: https://blog.csdn.net/qq21497936/article/details/108648385
Readers, the knowledge is endless but the manpower is poor. Either change your needs, find a professional, or research by yourself

Red Fatty (Red Imitation)'s blog post: development technology collection (including Qt practical technology, Raspberry Pi, 3D, OpenCV, OpenGL, ffmpeg, OSG, single-chip microcomputer, software and hardware combination, etc.) is continuously updated... (click on the portal)

FFmpeg and SDL development column (click on the portal)

Previous: " FFmpeg Development Notes (5): Detailed explanation of the basic process of ffmpeg decoding (ffmpeg3 new decoding api) "
Next: Stay tuned

Preface

  After ffmpeg is decoded, the display needs to be synchronized. One is to display. This article uses SDL to display, and the other is to synchronize the timestamp.

FFmpeg decoding

  For the basic process of FFmpeg decoding, please refer to:
  " FFmpeg Development Notes (4): Detailed explanation of the basic process of ffmpeg decoding "
  " FFmpeg Development Notes (5): Detailed explanation of the basic process of ffmpeg decoding (ffmpeg3 new decoding api) "

SDL display

  For the basic process of SDL displaying pictures, please refer to:
  " SD Development Notes (3): Using SDL to render window colors and pictures "

ffmpeg synchronization

  ffmpeg synchronization includes audio, video, subtitles, etc. The synchronization described here is the synchronization of time and video display.

Basic process

  

Synchronization key

  Calculate the frame rate. After getting the information, use the stream context to get the total time and total number of frames. The time interval calculated in this way is the most accurate. You can also use the time base to calculate it. The accuracy is just a bit more complicated.
  Video and time synchronization, such as 25fps, then 40ms to display a frame, then calculate the time interval, hang before the time node of the next frame until it reaches the time node of the frame display.
  Message-based is better processed, calculate the time interval of the next frame, and display directly after this time point.
  Frame skipping may also be involved here. If the system is stuck below, the decoding time interval is longer. If the next frame is calculated according to the previous frame plus 40ms, the first stitch will be 0ms and the second stitch will be stuck. It is 50ms, and the third needle is 90ms, so the overall time is lengthened.
  In summary, when ffmpeg is doing synchronous display, it needs to select the initial playback time as the reference, and use the frame number and frame interval to calculate the display time of the next frame.

Detailed explanation of ffmpeg synchronization related structure

AVStream

  The structure of the stream information, which contains the instance pointer of AVCodecContext

struct AVStream {
    AVCodecContext *codec;     // 编码器相关的信息
    AVRational avg_frame_rate; // 根据该参数,可以把PTS转化为实际的时间(单位为秒s)
    int64_t duration;          // 视频时长，单位为10ms，不是ms，所以要除以10000
    int64_t nb_frames;         // 视频总帧数
}

AVCodecContext

  This structure is the encoding context information. After the file stream is probed, the specific relevant information of the file stream can be obtained, and the relevant information about the encoding and decoding is in this file structure.
  The variables related to the synchronous video display are explained in detail here. For others, you can see the definition of the structure AVCodecContext on the ffmpeg source code.

struct AVCodecContext {
    AVMediaType codec_type;        // 编码器的类型，如视频、音频、字幕等等
    AVCodec *codec；               // 使用的编码器
    int bit_rata;                  // 平均比特率
    AVRational time_base：         // 根据该参数，可以把PTS转化为实际的时间(单位为秒s)
    int width, height：            // 如果是视频的话，代表宽和高
    enum AVPixelFormat pix_fmt;    // 代表视频像素的格式...
} AVCodecContext;

  

  

Demo source code

void FFmpegManager::testDecodeSyncShow()
{
//    QString fileName = "test/1.avi";
    QString fileName = "test/1.mp4";


    // SDL相关变量预先定义
    SDL_Window *pSDLWindow = 0;
    SDL_Renderer *pSDLRenderer = 0;
    SDL_Surface *pSDLSurface = 0;
    SDL_Texture *pSDLTexture = 0;
    SDL_Event event;

    qint64 startTime = 0;                           // 记录播放开始
    int currentFrame = 0;                           // 当前帧序号
    double fps = 0;                                 // 帧率
    double interval = 0;                            // 帧间隔

    // ffmpeg相关变量预先定义与分配
    AVFormatContext *pAVFormatContext = 0;          // ffmpeg的全局上下文，所有ffmpeg操作都需要
    AVStream *pAVStream = 0;                        // ffmpeg流信息
    AVCodecContext *pAVCodecContext = 0;            // ffmpeg编码上下文
    AVCodec *pAVCodec = 0;                          // ffmpeg编码器
    AVPacket *pAVPacket = 0;                        // ffmpag单帧数据包
    AVFrame *pAVFrame = 0;                          // ffmpeg单帧缓存
    AVFrame *pAVFrameRGB32 = 0;                     // ffmpeg单帧缓存转换颜色空间后的缓存
    struct SwsContext *pSwsContext = 0;             // ffmpag编码数据格式转换

    int ret = 0;                                    // 函数执行结果
    int videoIndex = -1;                            // 音频流所在的序号
    int numBytes = 0;                               // 解码后的数据长度
    uchar *outBuffer = 0;                           // 解码后的数据存放缓存区

    pAVFormatContext = avformat_alloc_context();    // 分配
    pAVPacket = av_packet_alloc();                  // 分配
    pAVFrame = av_frame_alloc();                    // 分配
    pAVFrameRGB32 = av_frame_alloc();               // 分配
    if(!pAVFormatContext || !pAVPacket || !pAVFrame || !pAVFrameRGB32)
    {
        LOG << "Failed to alloc";
        goto END;
    }
    // 步骤一：注册所有容器和编解码器（也可以只注册一类，如注册容器、注册编码器等）
    av_register_all();

    // 步骤二：打开文件(ffmpeg成功则返回0)
    LOG << "文件:" << fileName << "，是否存在：" << QFile::exists(fileName);
    ret = avformat_open_input(&pAVFormatContext, fileName.toUtf8().data(), 0, 0);
    if(ret)
    {
        LOG << "Failed";
        goto END;
    }
    // 步骤三：探测流媒体信息
    ret = avformat_find_stream_info(pAVFormatContext, 0);
    if(ret < 0)
    {
        LOG << "Failed to avformat_find_stream_info(pAVFormatContext, 0)";
        goto END;
    }
    // 步骤四：提取流信息,提取视频信息
    for(int index = 0; index < pAVFormatContext->nb_streams; index++)
    {
        pAVCodecContext = pAVFormatContext->streams[index]->codec;
        pAVStream = pAVFormatContext->streams[index];
        switch (pAVCodecContext->codec_type)
        {
        case AVMEDIA_TYPE_UNKNOWN:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_UNKNOWN";
            break;
        case AVMEDIA_TYPE_VIDEO:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_VIDEO";
            videoIndex = index;
            LOG;
            break;
        case AVMEDIA_TYPE_AUDIO:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_AUDIO";
            break;
        case AVMEDIA_TYPE_DATA:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_DATA";
            break;
        case AVMEDIA_TYPE_SUBTITLE:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_SUBTITLE";
            break;
        case AVMEDIA_TYPE_ATTACHMENT:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_ATTACHMENT";
            break;
        case AVMEDIA_TYPE_NB:
            LOG << "流序号:" << index << "类型为:" << "AVMEDIA_TYPE_NB";
            break;
        default:
            break;
        }
        // 已经找打视频品流
        if(videoIndex != -1)
        {
            break;
        }
    }

    if(videoIndex == -1 || !pAVCodecContext)
    {
        LOG << "Failed to find video stream";
        goto END;
    }

    // 步骤五：对找到的视频流寻解码器
    pAVCodec = avcodec_find_decoder(pAVCodecContext->codec_id);
    if(!pAVCodec)
    {
        LOG << "Fialed to avcodec_find_decoder(pAVCodecContext->codec_id):"
            << pAVCodecContext->codec_id;
        goto END;
    }
    // 步骤六：打开解码器
    ret = avcodec_open2(pAVCodecContext, pAVCodec, NULL);
    if(ret)
    {
        LOG << "Failed to avcodec_open2(pAVCodecContext, pAVCodec, pAVDictionary)";
        goto END;
    }

    // 显示视频相关的参数信息（编码上下文）
    LOG << "比特率:" << pAVCodecContext->bit_rate;

    LOG << "宽高:" << pAVCodecContext->width << "x" << pAVCodecContext->height;
    LOG << "格式:" << pAVCodecContext->pix_fmt;
    LOG << "帧率分母:" << pAVCodecContext->time_base.den;
    LOG << "帧率分子:" << pAVCodecContext->time_base.num;
    LOG << "帧率分母:" << pAVStream->avg_frame_rate.den;
    LOG << "帧率分子:" << pAVStream->avg_frame_rate.num;
    LOG << "总时长:" << pAVStream->duration / 10000.0 << "s";
    LOG << "总帧数:" << pAVStream->nb_frames;
    fps = pAVStream->nb_frames / (pAVStream->duration / 10000.0);
    LOG << "平均帧率:" << fps;
    interval = pAVStream->duration / 10.0 / pAVStream->nb_frames;
    LOG << "帧间隔:" << interval << "ms";
    // 步骤七：对拿到的原始数据格式进行缩放转换为指定的格式高宽大小
    pSwsContext = sws_getContext(pAVCodecContext->width,
                                 pAVCodecContext->height,
                                 pAVCodecContext->pix_fmt,
                                 pAVCodecContext->width,
                                 pAVCodecContext->height,
                                 AV_PIX_FMT_RGBA,
                                 SWS_FAST_BILINEAR,
                                 0,
                                 0,
                                 0);
    numBytes = avpicture_get_size(AV_PIX_FMT_RGBA,
                                  pAVCodecContext->width,
                                  pAVCodecContext->height);
    outBuffer = (uchar *)av_malloc(numBytes);
    // pAVFrame32的data指针指向了outBuffer
    avpicture_fill((AVPicture *)pAVFrameRGB32,
                   outBuffer,
                   AV_PIX_FMT_RGBA,
                   pAVCodecContext->width,
                   pAVCodecContext->height);


    ret = SDL_Init(SDL_INIT_VIDEO);
    if(ret)
    {
        LOG << "Failed";
        goto END;
    }
    pSDLWindow = SDL_CreateWindow(fileName.toUtf8().data(),
                                  0,
                                  0,
                                  pAVCodecContext->width,
                                  pAVCodecContext->height,
                                  SDL_WINDOW_ALWAYS_ON_TOP | SDL_WINDOW_OPENGL);
    if(!pSDLWindow)
    {
        LOG << "Failed";
        goto END;
    }
    pSDLRenderer = SDL_CreateRenderer(pSDLWindow, -1, 0);
    if(!pSDLRenderer)
    {
        LOG << "Failed";
        goto END;
    }

    startTime = QDateTime::currentDateTime().toMSecsSinceEpoch();
    currentFrame = 0;

    // 步骤八：读取一帧数据的数据包
    while(av_read_frame(pAVFormatContext, pAVPacket) >= 0)
    {
        if(pAVPacket->stream_index == videoIndex)
        {
            // 步骤八：对读取的数据包进行解码
            ret = avcodec_send_packet(pAVCodecContext, pAVPacket);
            if(ret)
            {
                LOG << "Failed to avcodec_send_packet(pAVCodecContext, pAVPacket) ,ret =" << ret;
                break;
            }
            while(!avcodec_receive_frame(pAVCodecContext, pAVFrame))
            {
                sws_scale(pSwsContext,
                          (const uint8_t * const *)pAVFrame->data,
                          pAVFrame->linesize,
                          0,
                          pAVCodecContext->height,
                          pAVFrameRGB32->data,
                          pAVFrameRGB32->linesize);
                // 格式为RGBA=8:8:8:8”
                // rmask 应为 0xFF000000  但是颜色不对 改为 0x000000FF 对了
                // gmask     0x00FF0000                  0x0000FF00
                // bmask     0x0000FF00                  0x00FF0000
                // amask     0x000000FF                  0xFF000000
                // 测试了ARGB，也是相反的，而QImage是可以正确加载的
                // 暂时只能说这个地方标记下，可能有什么设置不对什么的
                pSDLSurface = SDL_CreateRGBSurfaceFrom(outBuffer,
                                                       pAVCodecContext->width,
                                                       pAVCodecContext->height,
                                                       4 * 8,
                                                       pAVCodecContext->width * 4,
                                                       0x000000FF,
                                                       0x0000FF00,
                                                       0x00FF0000,
                                                       0xFF000000
                                                       );
                pSDLTexture = SDL_CreateTextureFromSurface(pSDLRenderer, pSDLSurface);

                SDL_FreeSurface(pSDLSurface);
//                pSDLSurface = SDL_LoadBMP("testBMP/1.bmp");
//                pSDLTexture = SDL_CreateTextureFromSurface(pSDLRenderer, pSDLSurface);

                // 清除Renderer
                SDL_RenderClear(pSDLRenderer);
                // Texture复制到Renderer
                SDL_RenderCopy(pSDLRenderer, pSDLTexture, 0, 0);
                // 更新Renderer显示
                SDL_RenderPresent(pSDLRenderer);
                // 事件处理
                SDL_PollEvent(&event);

            }
            // 下一帧
            currentFrame++;
            while(QDateTime::currentDateTime().toMSecsSinceEpoch() - startTime < currentFrame * interval)
            {
                SDL_Delay(1);
            }
            LOG << "current:" << currentFrame <<"," << time << (QDateTime::currentDateTime().toMSecsSinceEpoch() - startTime);
        }
    }
END:
    LOG << "释放回收资源";
    if(outBuffer)
    {
        av_free(outBuffer);
        outBuffer = 0;
    }
    if(pSwsContext)
    {
        sws_freeContext(pSwsContext);
        pSwsContext = 0;
        LOG << "sws_freeContext(pSwsContext)";
    }
    if(pAVFrameRGB32)
    {
        av_frame_free(&pAVFrameRGB32);
        pAVFrame = 0;
        LOG << "av_frame_free(pAVFrameRGB888)";
    }
    if(pAVFrame)
    {
        av_frame_free(&pAVFrame);
        pAVFrame = 0;
        LOG << "av_frame_free(pAVFrame)";
    }
    if(pAVPacket)
    {
        av_free_packet(pAVPacket);
        pAVPacket = 0;
        LOG << "av_free_packet(pAVPacket)";
    }
    if(pAVCodecContext)
    {
        avcodec_close(pAVCodecContext);
        pAVCodecContext = 0;
        LOG << "avcodec_close(pAVCodecContext);";
    }
    if(pAVFormatContext)
    {
        avformat_close_input(&pAVFormatContext);
        avformat_free_context(pAVFormatContext);
        pAVFormatContext = 0;
        LOG << "avformat_free_context(pAVFormatContext)";
    }

    // 步骤五：销毁渲染器
    SDL_DestroyRenderer(pSDLRenderer);
    // 步骤六：销毁窗口
    SDL_DestroyWindow(pSDLWindow);
    // 步骤七：退出SDL
    SDL_Quit();
}

Project template v1.2.0

  Corresponding project template v1.2.0: Add decoded video and use SDL to display Demo.

Previous: " FFmpeg Development Notes (5): Detailed explanation of the basic process of ffmpeg decoding (ffmpeg3 new decoding api) "
Next: Stay tuned

Original blogger’s blog address: https://blog.csdn.net/qq21497936
Original blogger’s blog navigation: https://blog.csdn.net/qq21497936/article/details/102478062
This article’s blog address: https://blog .csdn.net/qq21497936/article/details/108648385