Audio and video development tour (60) - Debugging and analysis of common structures of FFmpeg (decapsulation part)

ffplay breakpoint debugging
(Decapsulation part) Analysis of common structures and the relationship between them
material
reward

If a worker wants to do a good job, he must first sharpen his tools. Breakpoint debugging is very important for us to sort out the process and troubleshoot problems. The debugging of ffmpeg can be conveniently debugged and analyzed on IDEs such as XCode, VS code, and QT. In this article, we take XCode as an example to introduce the breakpoint debugging of ffplay, and use ffmpeg4.4 version for analysis.

1. Breakpoint debugging of ffplay

First download and compile ffmpeg. For details, please refer to Audio and Video Development Tour (33) -
The difference between FFmpeg (3.x and 4.x) used by cross-compiling android is that we are not cross-compiling this time, but compiling and installing on Mac debugging.

./configure --enable-static --disable-shared --enable-debug --disable-doc --disable-x86asm --enable-nonfree  --enable-libvpx --enable-gpl  --enable-opengl --enable-libx264  --enable-libx265 --enable-libvmaf
make -j8
sudo make install

After the compilation is successful, we will see several important executable files ffmpeg_g、ffprobe_g以及ffplay_g, and they will be used in the next operation and debugging.
How to configure and debug ffmpeg source code under Xcode, please refer to: Xcode Debugging ffmpeg Source Code (15) - Brief Book

We analyze the structure used in the ffplay decapsulation (read_thread) process at the breakpoint of the main function of ffplay.c.

open media stream

VideoState *stream_open(const char *filename,const AVInputFormat *iformat)

Involved in the structure: AVInputFormat

Start readthread to start reading

    is->read_tid     = SDL_CreateThread(read_thread, "read_thread", is);

Allocate AVFormatContext memory

 AVFormatContext   ic = avformat_alloc_context();

open streaming file

int avformat_open_input(AVFormatContext **ps, const char *filename,
                        const AVInputFormat *fmt, AVDictionary **options)

Structures involved: AVFormatContext, AVInputFormat, AVDictionary

get stream information

int avformat_find_stream_info(AVFormatContext *ic, AVDictionary **options)

Involved in the structure: AVStream AVCodecParameters AVRational

Loop read frame data

 for (;;) {
     ...
     int av_read_frame(AVFormatContext *s, AVPacket *pkt)
     ...
}

Involving structures: AVFormatContext, AVPacket, etc.

The decapsulation process comes here first. It can be seen that if you want to learn the source code of ffplay, you must first figure out the main process and the key structures involved in the process.
In the next section, we will analyze these structures in detail.

3. (Decapsulation part) common structures and relationship analysis between common structures and relationship analysis

3.1 Commonly used structures and the relationship between them

FFMPEG中结构体很多。最关键的结构体可以分成以下几类：
a)        解协议（http,rtsp,rtmp,mms）

AVIOContext，URLProtocol，URLContext主要存储视音频使用的协议的类型以及状态。URLProtocol存储输入视音频使用的封装格式。每种协议都对应一个URLProtocol结构。（注意：FFMPEG中文件也被当做一种协议“file”）

b)        解封装（flv,avi,rmvb,mp4）

AVFormatContext主要存储视音频封装格式中包含的信息；AVInputFormat存储输入视音频使用的封装格式。每种视音频封装格式都对应一个AVInputFormat 结构。

c)        解码（h264,mpeg2,aac,mp3）

每个AVStream存储一个视频/音频流的相关数据；每个AVStream对应一个AVCodecContext，存储该视频/音频流使用解码方式的相关数据；每个AVCodecContext中对应一个AVCodec，包含该视频/音频对应的解码器。每种解码器都对应一个AVCodec结构。

d) 存数据

视频的话，每个结构一般是存一帧；音频可能有好几帧

解码前数据：AVPacket

解码后数据：AVFrame


引用自： https://blog.csdn.net/leixiaohua1020/article/details/11693997

The relationship between them is as follows:

Picture from: The relationship between the most critical structures in FFMPEG

3.2. The structure of AVFormatContext
is defined in libavformat/Avformat.h. It is a data structure throughout, and many functions use it as a parameter. The role of several main variables is as follows:

struct AVInputFormat *iformat：输入数据的封装格式
struct AVOutputFormat *oformat：输出数据的封装格式

AVIOContext *pb：输入数据的缓存

unsigned int nb_streams：视音频流的个数

AVStream **streams：视音频流

char filename[1024]：文件名

int64_t duration：时长（单位：微秒us，转换为秒需要除以1000000）

int bit_rate：比特率（单位bps，转换为kbps需要除以1000）

AVDictionary *metadata：元数据

3.3 AVInputFormat
The definition of this structure is also in libavformat/Avformat.h, which is the main variable of the decapsulator object. The function is as follows

const char *name: 格式的名称
const char *mime_type： mime类型如 video/avc video/hevc audio/aac等

以及一系列函数指针
int (*read_probe)(const AVProbeData *);
int (*read_packet)(struct AVFormatContext *, AVPacket *pkt);
int (*read_close)(struct AVFormatContext *);
int (*read_seek)(struct AVFormatContext *,
                     int stream_index, int64_t timestamp, int flags);
int (*read_play)(struct AVFormatContext *);

int (*read_pause)(struct AVFormatContext *);
int (*read_seek2)(struct AVFormatContext *s, int stream_index, int64_t min_ts, int64_t ts, int64_t max_ts, int flags);

3.4 AVStream
Each AVStream stores the relevant data of a video/audio stream; it is the stream object separated by the decapsulator, that is, the product of decapsulation, which is stored in the AVFormatcontext.

The structure definition is also in libavformat/Avformat.h, the main variables are as follows:

int index;  流索引
int id; 流id
void *priv_data; 流数据
AVRational time_base; 时间基，通过该值可以把PTS，DTS转化为真正的时间；PTS*time_base=真正的时间
int64_t duration：流长度
AVRational sample_aspect_ratio； 采样率
AVRational avg_frame_rate：帧率
AVCodecContext *codec：指向该视频/音频流的AVCodecContext（它们是一一对应的关系）

AVStream is the output of the decapsulation link, and it is also the input of the decoding link. Each AVStream corresponds to an AVCodecContext, which stores the relevant data of the decoding method used by the video/audio stream; each AVCodecContext corresponds to an AVCodec, including the video/audio corresponding decoder. Each decoder corresponds to an AVCodec structure.
The data structure analysis of the decoding part will be analyzed and studied in the next article.

3.5 AVPacket
stores the structure of compressed and encoded data-related information, saves the data after decapsulation and before decoding, and information such as PTS, DTS, Duration, and streamId. The definition of this structure is
located in libavcodec/Packet.h, and the main variables are as follows:

  uint8_t *data; 对于H.264来说。1个AVPacket的data通常对应一个NAL。
int   size：data的大小
int64_t pts：显示时间戳
int64_t dts：解码时间戳
AVPacketSideData *side_data;附加信息

3. Information

"Android Audio and Video Development" - Chapter 8
Xcode Debugging ffmpeg Source Code (15)
The relationship between the most critical structures in FFMPEG
FFMPEG structure analysis: AVFormatContext
FFMPEG structure analysis: AVStream
FFMPEG structure analysis: AVPacket

4. Harvest

Through the study and practice of this article, we have learned

How to debug ffmpeg with breakpoints under Xcode and analyze the ffplay decapsulation process
Understand the relationship between commonly used structures: unpack, decapsulate, decode the corresponding structure and the relationship between them
Understand the main variables and functions of several key structures related to decapsulation. AVFormatContext, AVInputFormat, AVStream

Thank you for reading.
In the next article, we analyze the common structure of ffmpeg decoding part. Welcome to pay attention to the official account "Audio and Video Development Journey" and learn and grow together.
Welcome to exchange