1. Experimental purpose
On the basis of mastering FFMPEG encoding, further master the process of FFMPEG video decoding, including the configuration of the development host environment and cloud platform, understanding of the video decoding program, compilation and operation of the code, etc. Focus on mastering the commonly used API function interfaces, commonly used structures, basic decoding processes, and key steps in the FFMPEG decoding process.
2. Experimental content
Build an experimental development environment, compile and run the decoding program, and decode the encoded video stream.
3. Development environment
Development host: Ubuntu 22.04 LTS
Hardware: Suaneng SE5
4. Experimental equipment
Development host + cloud platform (or SE5 hardware)
5. Experimental process and conclusion
5.1 FFMPEG decoding principle and process
FFMPEG also provides an interface for decoding video compressed files, supporting decoding of H264 , H265 , MJPEG and other video files. Supports decoding and outputting YUV files. FFMPEG not only supports decoding video compressed files, but also supports decoding audio files. For example, audio AAC frames are decoded into PCM data through the decoder .
The basic process of FFMPEG decoding is similar to FFMPEG encoding in the initialization part, but during the decoding process the av_read_frame and avcodec_decode_video2 functions are mainly called for decoding. Among them, av_read_frame loops to read one frame of video frame data from the cache, and the avcodec_decode_video2 function is responsible for decoding one frame of compressed data. The basic process of standard FFMPEG decoding is shown in the figure below:
It should be noted that since the FFMPEG interface of computing uses hardware for acceleration, in PCIE mode it is necessary to specify an accelerator card and perform memory synchronization and other operations. Please refer to the following code:
#ifdef BM_PCIE_MODE
av_dict_set_int(&opts, "zero_copy", pcie_no_copyback, 0);
av_dict_set_int(&opts, "sophon_idx", sophon_idx, 0);
#endif
if(output_format_mode == 101)
av_dict_set_int(&opts, "output_format", output_format_mode, 18);
//if(extra_frame_buffer_num > 5)
av_dict_set_int(&opts, "extra_frame_buffer_num", extra_frame_buffer_num, 0);
//av_dict_set_int(&opts, "extra_frame_buffer_num", 1, 0);5.2 FFMPEG decoding key functions
Main function:
FILE *fp_yuv = fopen(output_file.data(), "wb+");
av_log_set_level(AV_LOG_DEBUG); // set debug level
reader.openDec(input_file.data(), 1, "h264_bm", 100, 60, 0, 0);
pFormatCtx = reader.ifmt_ctx;
videoindex = -1;
for (i = 0; i < pFormatCtx->nb_streams; i++) {
if (pFormatCtx->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO)
{
videoindex = i;
break;
}
}
if (videoindex == -1)
{
printf("Didn't find a video stream.\n");
return -1;
}
cout << "video index " << videoindex << endl;
pCodecCtx = reader.video_dec_ctx;
pCodec = reader.decoder;
pFrame = av_frame_alloc();
pFrameYUV = av_frame_alloc();
out_buffer = (uint8_t *)av_malloc(avpicture_get_size(AV_PIX_FMT_YUV420P, pCodecCtx->width, pCodecCtx->height));
avpicture_fill((AVPicture *)pFrameYUV, out_buffer, AV_PIX_FMT_YUV420P, pCodecCtx->width, pCodecCtx->height);
packet = (AVPacket *)av_malloc(sizeof(AVPacket));
av_dump_format(pFormatCtx, 0, input_file.data(), 0);
img_convert_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, AV_PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL);
long long framecount = 0;
while (av_read_frame(pFormatCtx, packet) >= 0)
{ //读取一帧压缩数据
if (packet->stream_index == videoindex)
{
//解码一帧压缩数据
ret = avcodec_decode_video2(pCodecCtx, pFrame, &got_picture, packet);
if (ret < 0)
{
printf("Decode Error.\n");
return -1;
}
if (got_picture)
{
sws_scale(img_convert_ctx, (const uint8_t *const *)pFrame->data, pFrame->linesize, 0, pCodecCtx->height,pFrameYUV->data, pFrameYUV->linesize);
y_size = pCodecCtx->width * pCodecCtx->height;
fwrite(pFrameYUV->data[0], 1, y_size, fp_yuv); // Y
fwrite(pFrameYUV->data[1], 1, y_size / 4, fp_yuv); // U
fwrite(pFrameYUV->data[2], 1, y_size / 4, fp_yuv); // V
printf("\rfinish %lld [%c].", ++framecount, progressbar_icon[framecount % 12]);
fflush(stdout);
}
}
av_free_packet(packet);
}
// flush decoder
/*当av_read_frame()循环退出的时候，实际上解码器中可能还包含剩余的几帧数据。
因此需要通过“flush_decoder”将这几帧数据输出。
“flush_decoder”功能简而言之即直接调用avcodec_decode_video2()获得AVFrame，而不再向解码器传递AVPacket。*/
while (1)
{
ret = avcodec_decode_video2(pCodecCtx, pFrame, &got_picture, packet);
if (ret < 0)
break;
if (!got_picture)
break;
sws_scale(img_convert_ctx, (const uint8_t *const *)pFrame->data, pFrame->linesize, 0, pCodecCtx->height,
pFrameYUV->data, pFrameYUV->linesize);
int y_size = pCodecCtx->width * pCodecCtx->height;
//输出到文件
fwrite(pFrameYUV->data[0], 1, y_size, fp_yuv); // Y
fwrite(pFrameYUV->data[1], 1, y_size / 4, fp_yuv); // U
fwrite(pFrameYUV->data[2], 1, y_size / 4, fp_yuv); // V
printf("\rfinish %lld [%c].",++framecount, progressbar_icon[framecount % 12]);
fflush(stdout);
}
sws_freeContext(img_convert_ctx);
//关闭文件以及释放内存
fclose(fp_yuv);
cout << "Total Decode " << framecount << " frames" << endl;
av_frame_free(&pFrameYUV);
av_frame_free(&pFrame);
From the above code, we can find that openDec is called here to start the decoder, and then the decoded video frame is obtained through grabFrame , which actually completes the decoding action. Both interfaces are provided through the VideoDec_FFMPEG class. Next we introduce how to write the VideoDec_FFMPEG class, which is also the core of this example.
VideoDec_FFMPEG class
Initialization constructor:
VideoDec_FFMPEG::VideoDec_FFMPEG()
{
ifmt_ctx = NULL;
video_dec_ctx = NULL;
video_dec_par = NULL;
decoder = NULL;
width = 0;
height = 0;
pix_fmt = 0;
video_stream_idx = -1;
refcount = 1;
av_init_packet(&pkt);
pkt.data = NULL;
pkt.size = 0;
frame = av_frame_alloc();
}Destructor:
VideoDec_FFMPEG::~VideoDec_FFMPEG()
{
closeDec();
printf("#VideoDec_FFMPEG exit \n");
}Key function openDec:
int VideoDec_FFMPEG::openDec(const char* filename,int codec_name_flag,
const char *coder_name,int output_format_mode,
int extra_frame_buffer_num,int sophon_idx, int pcie_no_copyback)//文件的打开，解码器的初始化
{
int ret = 0;
AVDictionary *dict = NULL;
av_dict_set(&dict, "rtsp_flags", "prefer_tcp", 0);
//打开媒体流
ret = avformat_open_input(&ifmt_ctx, filename, NULL, &dict);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Cannot open input file\n");
return ret;
}
//获取媒体信息
ret = avformat_find_stream_info(ifmt_ctx, NULL);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Cannot find stream information\n");
return ret;
}
//打开编码器（二次封装函数）
ret = openCodecContext(&video_stream_idx, &video_dec_ctx, ifmt_ctx, AVMEDIA_TYPE_VIDEO,codec_name_flag, coder_name, output_format_mode, extra_frame_buffer_num);
if (ret >= 0) {
width = video_dec_ctx->width;
height = video_dec_ctx->height;
pix_fmt = video_dec_ctx->pix_fmt;
}
av_log(video_dec_ctx, AV_LOG_INFO,
"openDec video_stream_idx = %d, pix_fmt = %d\n",video_stream_idx, pix_fmt);
av_dict_free(&dict);
return ret;
}From the above code, we can find that the media stream is opened through avformat_open_input and the media information is obtained through the avformat_find_stream_info function. And called openCodecContext to open the encoder. Here openCodecContext is a secondary encapsulation function, and the implementation method is as follows:
int VideoDec_FFMPEG::openCodecContext(int *stream_idx,AVCodecContext **dec_ctx, AVFormatContext *fmt_ctx, enum AVMediaType type,int codec_name_flag, const char *coder_name, int output_format_mode,int extra_frame_buffer_num, int sophon_idx, int pcie_no_copyback)
{
int ret, stream_index;
AVStream *st;
AVCodec *dec = NULL;
AVDictionary *opts = NULL;
ret = av_find_best_stream(fmt_ctx, type, -1, -1, NULL, 0);
if (ret < 0) {
av_log(NULL, AV_LOG_ERROR, "Could not find %s stream\n", av_get_media_type_string(type));
return ret;
}
stream_index = ret;
st = fmt_ctx->streams[stream_index];
/* find decoder for the stream */
if(codec_name_flag && coder_name)
decoder = findBmDecoder((AVCodecID)0,coder_name,codec_name_flag, AVMEDIA_TYPE_VIDEO);
else
decoder = findBmDecoder(st->codecpar->codec_id);
if (!decoder) {
av_log(NULL, AV_LOG_FATAL,"Failed to find %s codec\n",
av_get_media_type_string(type));
return AVERROR(EINVAL);
}
/* Allocate a codec context for the decoder */
*dec_ctx = avcodec_alloc_context3(decoder);
if (!*dec_ctx) {
av_log(NULL, AV_LOG_FATAL, "Failed to allocate the %s codec context\n",
av_get_media_type_string(type));
return AVERROR(ENOMEM);
}
/* Copy codec parameters from input stream to output codec context */
ret = avcodec_parameters_to_context(*dec_ctx, st->codecpar);
if (ret < 0) {
av_log(NULL, AV_LOG_FATAL, "Failed to copy %s codec parameters to decoder context\n", av_get_media_type_string(type));
return ret;
}
video_dec_par = st->codecpar;
/* Init the decoders, with or without reference counting */
av_dict_set(&opts, "refcounted_frames", refcount ? "1" : "0", 0);
if(output_format_mode == 101)
av_dict_set_int(&opts, "output_format", output_format_mode, 18);
av_dict_set_int(&opts, "extra_frame_buffer_num", extra_frame_buffer_num, 0);
ret = avcodec_open2(*dec_ctx, dec, &opts);
if (ret < 0) {
av_log(NULL, AV_LOG_FATAL, "Failed to open %s codec\n",
av_get_media_type_string(type));
return ret;
}
*stream_idx = stream_index;
av_dict_free(&opts);
return 0;
}From the above code, we can find that in the process of opening the encoder , openCodecContext mainly uses findBmDecoder to find the encoder (secondary encapsulation function), avcodec_alloc_context3 allocates the encoding context , avcodec_parameters_to_context sets the encoder and uses the av_dict_set_int function to set some parameters. Finally, open the encoder through avcodec_open2 .
Key function grabFrame: decode video frame
AVFrame * VideoDec_FFMPEG::grabFrame()//返回一帧解码的结果
{
int ret = 0;
int got_frame = 0;
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);
while (1) {
av_packet_unref(&pkt);
ret = av_read_frame(ifmt_ctx, &pkt);
if (ret < 0) {
if (ret == AVERROR(EAGAIN)) {
gettimeofday(&tv2, NULL);
if(((tv2.tv_sec - tv1.tv_sec) * 1000 + (tv2.tv_usec - tv1.tv_usec) / 1000) > 1000*60) {
av_log(video_dec_ctx, AV_LOG_WARNING, "av_read_frame failed ret(%d) retry time >60s.\n", ret);
break;
}
usleep(10*1000);
continue;
}
av_log(video_dec_ctx, AV_LOG_ERROR, "av_read_frame ret(%d) maybe eof...\n", ret);
return NULL; // TODO
}
if (pkt.stream_index != video_stream_idx) {
continue;
}
if (!frame) {
av_log(video_dec_ctx, AV_LOG_ERROR, "Could not allocate frame\n");
return NULL;
}
if (refcount) {
av_frame_unref(frame);
}
gettimeofday(&tv1, NULL);
ret = avcodec_decode_video2(video_dec_ctx, frame, &got_frame, &pkt);
if (ret < 0) {
av_log(video_dec_ctx, AV_LOG_ERROR, "Error decoding video frame (%d)\n", ret);
continue; // TODO
}
if (!got_frame) {
continue;
}
width = video_dec_ctx->width;
height = video_dec_ctx->height;
pix_fmt = video_dec_ctx->pix_fmt;
if (frame->width!= width||frame->height != height || frame->format != pix_fmt){
av_log(video_dec_ctx, AV_LOG_ERROR,
"Error: Width, height and pixel format have to be "
"constant in a rawvideo file, but the width, height or "
"pixel format of the input video changed:\n"
"old: width = %d, height = %d, format = %s\n"
"new: width = %d, height = %d, format = %s\n",
width, height, av_get_pix_fmt_name((AVPixelFormat)pix_fmt),
frame->width, frame->height,
av_get_pix_fmt_name((AVPixelFormat)frame->format));
continue;
}
break;
}
return frame;
Close the decoder after final use:
void VideoDec_FFMPEG::closeDec()
{
if (video_dec_ctx) {
avcodec_free_context(&video_dec_ctx);
video_dec_ctx = NULL;
}
if (ifmt_ctx) {
avformat_close_input(&ifmt_ctx);
ifmt_ctx = NULL;
}
if (frame) {
av_frame_free(&frame);
frame = NULL;
}
}5.3 Decoding experimental process
Generate executable file:
Follow the above experimental steps to generate an executable file and upload it to the computing embedded platform or cloud platform. The specific operations will not be described again. At this time, the files in the test folder are as shown in the figure.
root@d11ae417e206:/tmp/test# ls
ffmpeg_decoder test.h264
The test.h264 here can use the encoding file generated in Experiment 8.
Grant permissions to the executable file and execute it.
root@d11ae417e206:/tmp/test# chmod 777 ffmpeg_decoder
Run command:
After generating and uploading the compiled file, run it on the target development machine terminal according to the following instructions. The specific instruction parameter settings will be introduced in detail below.
./ffmpeg_decoder test.h264 out.yuv
The running results are as follows:
The parameter stream displayed in the running result interface represents the input code stream file, bm decoder id represents the decoder name used, sophon device represents the sophon chip serial number used in PCIE mode , bm output format represents the format of the output data, and mode bitstream represents Bit stream mode, frame delay indicates the number of frames delayed by the decoder, and pix_fmt indicates the pixel format supported by the decoder. The key parts are shown in green font below:
root@ab162899a93b:/tmp/tmp6l8uq_dw# ./ffmpeg_decoder test111.h264 out.yuv
[NULL @ 0x449a10] Opening 'test111.h264' for reading
[file @ 0x44a240] Setting default whitelist 'file,crypto'
[h264 @ 0x449a10] Format h264 probed with size=2048 and score=51
[h264 @ 0x449a10] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
[AVBSFContext @ 0x4521c0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[AVBSFContext @ 0x4521c0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[AVBSFContext @ 0x4521c0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x44acc0] nal_unit_type: 7(SPS), nal_ref_idc: 3
[h264 @ 0x44acc0] nal_unit_type: 8(PPS), nal_ref_idc: 3
[h264 @ 0x44acc0] nal_unit_type: 5(IDR), nal_ref_idc: 3
[h264 @ 0x44acc0] Format yuv420p chosen by get_format().
[h264 @ 0x44acc0] Reinit context to 1920x1088, pix_fmt: yuv420p
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 2
[h264 @ 0x44acc0] no picture
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x44acc0] Increasing reorder buffer to 2
[h264 @ 0x44acc0] no picture ooo
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0x44acc0] Increasing reorder buffer to 3
[h264 @ 0x44acc0] no picture ooo
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0x44acc0] no picture ooo
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 1
[h264 @ 0x44acc0] no picture
[h264 @ 0x44acc0] nal_unit_type: 1(Coded slice of a non-IDR picture), nal_ref_idc: 0
[h264 @ 0x449a10] After avformat_find_stream_info() pos: 520810 bytes read:520810 seeks:0 frames:101
[AVBSFContext @ 0x4521c0] The input looks like it is Annex B already
[h264_bm @ 0x44c170] Format nv12 chosen by get_format().
[h264_bm @ 0x44c170] ff_get_format: nv12.
[h264_bm @ 0x44c170] bmctx->hw_accel=0
[h264_bm @ 0x44c170] bm decoder id: 0
[h264_bm @ 0x44c170] bm output format: 0
[h264_bm @ 0x44c170] mode bitstream: 2, frame delay: -1
BMvidDecCreateW5 board id 0 coreid 0
libbmvideo.so addr : /system/lib/libbmvideo.so, name_len: 12
vpu firmware addr: /system/lib/vpu_firmware/chagall_dec.bin
VERSION=0, REVISION=213135
[h264_bm @ 0x44c170] perf: 0
[h264_bm @ 0x44c170] init options: mode, 2, frame delay, -1, output format, 0, extra frame buffer number: 5, extra_data_flag: 1
[h264_bm @ 0x44c170] openDec video_stream_idx = 0, pix_fmt = 23
video index 0
Input #0, h264, from 'test111.h264':
Duration: N/A, bitrate: N/A
Stream #0:0, 101, 1/1200000: Video: h264 (High), 1 reference frame, nv12(progressive, left), 1920x1080 (1920x1088), 0/1, 25 fps, 25 tbr, 1200k tbn, 50 tbc
finish 96 [|].[h264_bm @ 0x44c170] flush all frame in the decoder frame buffer
may be endof.. please check it.............
may be endof.. please check it.............
may be endof.. please check it.............
may be endof.. please check it.............
finish 101 [/].Total Decode 101 frames
[AVIOContext @ 0x42f6d0] Statistics: 520810 bytes read, 0 seeks
#VideoDec_FFMPEG exit
root@ab162899a93b:/tmp/tmp6l8uq_dw#