In the previous article, we analyzed and understood the common structures of the encapsulation part . In this article, we will learn and analyze the common structures of the decoding part.
Table of contents
- Breakpoint analysis ffplay decoding process and key structure
- (Decoding part) Analysis of common structures and the relationship between them
- material
- reward
1. Breakpoint analysis ffplay decoding process and key structure
or from read_threadthe analysis
stream_component_open(is, st_index[AVMEDIA_TYPE_AUDIO]);
ret = stream_component_open(is, st_index[AVMEDIA_TYPE_VIDEO]);
stream_component_open opens the specified stream
static int stream_component_open(VideoState *is, int stream_index)
AVFormatContext *ic = is->ic;
AVCodecContext *avctx;
const AVCodec *codec;
其中的关键函数如下
avctx = avcodec_alloc_context3(NULL);
int avcodec_parameters_to_context(AVCodecContext *codec,
const AVCodecParameters *par)
codec = avcodec_find_decoder(avctx->codec_id);
codec = avcodec_find_decoder_by_name(forced_codec_name);
//打开codec
avcodec_open2(avctx, codec, &opts)
//音频解码
decoder_init(&is->auddec, avctx, &is->audioq, is->continue_read_thread)
decoder_start(&is->auddec, audio_thread, "audio_decoder", is)
//视频解码
decoder_init(&is->viddec, avctx, &is->videoq, is->continue_read_thread)
decoder_start(&is->viddec, video_thread, "video_decoder", is)
avcodec_free_context(&avctx);
涉及到结构体：AVFormatContext、AVCodecContext、AVCodec
avcodec_parameters_to_context
int avcodec_parameters_to_context(AVCodecContext *codec,
const AVCodecParameters *par)
涉及到结构体：AVCodecContext、AVCodecParameters
Avcodec_find_decoder finds the decoder by codecid
const AVCodec *avcodec_find_decoder(enum AVCodecID id)
涉及到结构体：AVCodec
avcodec_open2
int avcodec_open2(AVCodecContext *avctx, const AVCodec *codec, AVDictionary **options)
涉及到结构体：AVCodecContext、AVCodec
decoding thread
static int audio_thread(void *arg)
static int video_thread(void *arg)
解码线程中以AVFrame作为解压缩后的结构体
涉及到结构体：AVFrame
Then start the decoding threads of audio and video respectively to start decoding. We can see that the main structures involved areAVCodecContext 、AVCodecParameters 、AVCodec 、AVFrame
As the decoded input AVPacket (a structure that stores information related to compressed and encoded data) and AVStream (each AVStream stores data related to a video/audio stream; it is a stream object separated by a decapsulator) we have already introduced in the previous article, Let's mainly analyze several structures related to decodingAVCodecContext 、AVCodec 、AVFrame
2. (Decoding part) Analysis of common structures and the relationship between them
2.1 Commonly used structures and the relationship between them (to quote again, Thor's summary is very good)
FFMPEG中结构体很多。最关键的结构体可以分成以下几类：
a) 解协议（http,rtsp,rtmp,mms）
AVIOContext，URLProtocol，URLContext主要存储视音频使用的协议的类型以及状态。URLProtocol存储输入视音频使用的封装格式。每种协议都对应一个URLProtocol结构。（注意：FFMPEG中文件也被当做一种协议“file”）
b) 解封装（flv,avi,rmvb,mp4）
AVFormatContext主要存储视音频封装格式中包含的信息；AVInputFormat存储输入视音频使用的封装格式。每种视音频封装格式都对应一个AVInputFormat 结构。
c) 解码（h264,mpeg2,aac,mp3）
每个AVStream存储一个视频/音频流的相关数据；每个AVStream对应一个AVCodecContext，存储该视频/音频流使用解码方式的相关数据；每个AVCodecContext中对应一个AVCodec，包含该视频/音频对应的解码器。每种解码器都对应一个AVCodec结构。
d) 存数据
视频的话，每个结构一般是存一帧；音频可能有好几帧
解码前数据：AVPacket
解码后数据：AVFrame
引用自： https://blog.csdn.net/leixiaohua1020/article/details/11693997
The relationship between them is as follows:
The relationship between the most critical structures in FFMPEG
2.2 AVCodecContext
AVCodecContext is a data structure describing the decoder context, which contains a lot of parameter information required by the encoder.
The structure definition is located in libavcodec/AVcodec.h, the main variables are as follows:
enum AVMediaType codec_type; // 该枚举定义在libavutil/Avutil.h中，编码的类型，音频、视频、字母等
const struct AVCodec *codec;//采用的解码器AVCodec，下面单独分析
enum AVCodecID codec_id; // 该枚举定义在libavcodec/Codec_id.h中，定了一了所有的编解码器id
void *priv_data;
struct AVCodecInternal *internal;//内部使用的上下文环境
void *opaque;
int64_t bit_rate;//平均码率
uint8_t *extradata; int extradata_size：针对特定编码器包含的附加信息（例如对于H.264解码器来说，存储SPS，PPS等）
AVRational time_base;//时间基，根据该参数，可以把PTS转化为实际的时间（单位为秒s）
int width, height;//仅视频类型用
int gop_size;//关键帧间隔
int max_b_frames;//最大b帧数量
int has_b_frames;//是否有b帧，关系到视频的压缩比率，一般b帧越多压缩比越大
int slice_count;//片的总和，关于slice相关，可以[音视频开发之旅（56) -H264/AVC基本结构](https://mp.weixin.qq.com/s?__biz=MzU5NjkxMjE5Mg==&mid=2247484355&idx=1&sn=538378561c16b640a4ea42bc1f354044&chksm=fe5a32ecc92dbbfa1d6a2e83f22aece727badb99966b6e621322ed8bf6b0cd8f0b2d1c262013&token=778944351&lang=zh_CN#rd)
AVRational sample_aspect_ratio;//采样率
/* audio only */
int sample_rate; ///< samples per second 音频采样率
int channels; ///< number of audio channels 通道数
enum AVSampleFormat sample_fmt; ///< sample format 采样格式,定义在libavutil/Samplefmt.h中
enum AVColorSpace colorspace;//颜色空间,定义在libavutil/Pixfmt.h
AVRational framerate;//帧率
enum AVPixelFormat sw_pix_fmt;//像素格式，如yuv420pdeng ,定义在libavutil/Pixfmt.h 如果设置不对导致解码器无法正常解码会出现花屏的情况。
There are many variables involved in this structure, and many of them are related to encoding. In fact, not too many are used for decoding.
2.3 AVCodec
AVCodec is a structure that stores encoder information.
The structure definition is located in libavcodec/Codec.h, the main variables are as follows:
const char *name;//codec的名称
enum AVMediaType type;//codec的类型，AVCodecContext中已经介绍
enum AVCodecID id;//codec的id，AVCodecContext中已经介绍
int capabilities;//编解码的能力,见 AV_CODEC_CAP_
const AVRational *supported_framerates;//支持的帧率
const enum AVPixelFormat *pix_fmts;//支持的像素格式
const int *supported_samplerates; //支持的音频采样率
const enum AVSampleFormat *sample_fmts;//支持的采样格式
下面是一些函数指针
int (*init)(struct AVCodecContext *);//初始化
int (*encode2)(struct AVCodecContext *avctx, struct AVPacket *avpkt,const struct AVFrame *frame, int *got_packet_ptr);//编码
int (*decode)(struct AVCodecContext *avctx, void *outdata,
int *got_frame_ptr, struct AVPacket *avpkt);
int (*close)(struct AVCodecContext *);//解码
int (*receive_packet)(struct AVCodecContext *avctx, struct AVPacket *avpkt);//接收packet数据
int (*receive_frame)(struct AVCodecContext *avctx, struct AVFrame *frame);//接收frame数据
void (*flush)(struct AVCodecContext *);//刷新缓冲区
2.4 AVFrame
AVFrame is generally used to store raw data (that is, uncompressed data, such as YUV, RGB for video, and PCM for audio), and also contains some related information. For example, data such as the macroblock type table, QP table, and motion vector table are stored during decoding. Relevant data is also stored during encoding.
The structure definition is located in libavutil/Frame.h, the main variables are as follows:
#define AV_NUM_DATA_POINTERS 8
uint8_t *data[AV_NUM_DATA_POINTERS];//对于planar格式的数据（例如YUV420P），则会分开成data[0]，data[1]，data[2]...（YUV420P中data[0]存Y，data[1]存U，data[2]存V）
int width, height;//视频宽高
int nb_samples;//每个信道音频采样点的个数
int format;//帧的像素格式
int key_frame;//1 -> keyframe, 0-> not
enum AVPictureType pict_type;//定义在libavutil/AVutil.h中，该帧的类型，I、P、B等
AVRational sample_aspect_ratio;//宽高比（16:9，4:3...）FFMPEG中用AVRational表达分数：
int64_t pts;//显示时间戳
int64_t pkt_dts;//从packet复制的PTS
int quality;
void *opaque;
int coded_picture_number;//编码帧序号
int display_picture_number;//显示帧序号
int8_t *qscale_table;//QP表 QP表指向一块内存，里面存储的是每个宏块的QP值。宏块的标号是从左往右，一行一行的来的。每个宏块对应1个QP。
uint8_t *mbskip_table;//跳过宏块表
int16_t (*motion_val[2])[2];//运动矢量表
int8_t *ref_index[2];//运动估计参考帧列表
int interlaced_frame;//交错帧，表示图像内容是交错的,即是否是隔行扫描
int sample_rate;//音频采样率
uint8_t motion_subsample_log2;//一个宏块中的运动矢量采样个数，取log2的.1个运动矢量所能代表的画面大小（用宽或者高表示，单位是像素），注意，这里取了log2。
3. Information
- "Android Audio and Video Development" - Chapter 8
- The relationship between the most critical structures in FFMPEG
- FFMPEG structure analysis: AVCodecContext
- FFMPEG structure analysis: AVCodec
- FFMPEG structure analysis: AVFrame
- FFMPEG realizes the conversion between YUV and RGB various image raw data (swscale)
4. Harvest
Through the study and practice of this article, the harvest is as follows:
- Breakpoint analysis and decoding process to deepen understanding
- Revisit the relationship between important structures of ffmpeg
- Understand the decoding-related structures AVCodecContext, AVCodec, and AVFrame. Among them, a lot of knowledge about encoding protocols is set, which requires systematic learning (X265)
Thank you for reading
the next article. The last article we learn about the key structure of fflay is to solve the related structure of the protocol. Welcome to pay attention to the official account "Audio and Video Development Journey" and learn and grow together.
Welcome to exchange