FFmpeg general introduction and memory model

1. Player framework

2. Common audio and video terms

• Container/File (Conainer/File): refers to multimedia files in a specific format, such as mp4, flv, mkv, etc.

• Media stream (Stream): represents a piece of continuous data on the time axis, such as a piece of audio data, a piece of video data or a piece of subtitle data, which can be compressed or uncompressed. Compressed data needs to be associated with a specific codec (Some stream audio is pure PCM).

• Data frame/data packet (Frame/Packet): Usually, a media stream is composed of a large number of data frames. For compressed data, the frame corresponds to the smallest processing unit of the codec, and data frames belonging to different media streams are interleaved stored in a container.

• Codec: The codec implements the conversion between compressed data and original data in units of frames.

3. Commonly used concept - multiplexer

4. Common concepts - codec

5. The overall structure of FFmpeg

6. FFMPEG has 8 commonly used libraries

• AVUtil: core tool library, many other modules below will rely on this library to do some basic audio and video processing operations.

• AVFormat: file format and protocol library, this module is one of the most important modules, it encapsulates the Protocol layer and the Demuxer and Muxer layers, making the protocol and format transparent to developers.

• AVCodec: Codec library, which encapsulates the Codec layer, but some Codecs have their own licenses. FFmpeg will not add libraries such as libx264 and FDK-AAC by default, but FFmpeg is like a platform, and other The third-party Codec is added as a plug-in, and then provides a unified interface for developers.

• AVFilter: audio and video filter library, this module provides processing including audio and video special effects, in the process of encoding and decoding using FFmpeg API, it is very convenient to directly use this module to do special effects processing for audio and video data A very efficient way.

• AVDevice: Input and output device library. For example, if you need to compile the tool ffplay for playing sound or video, you need to ensure that this module is opened, and you also need to pre-compile SDL, because the device module uses both sound and video playback. is the SDL library.

• SwrRessample: This module can be used for audio resampling, and can convert digital audio with various basic information such as the number of channels, data format, and sampling rate.

• SWScale: This module is a module for image format conversion. For example, YUV data can be converted to RGB data, and the zoom size is changed from 1280 720 to 800 480.

• PostProc: This module can be used for post-processing. When we use AVFilter, we need to turn on the switch of this module, because some basic functions of this module will be used in Filter.

7. Introduction to FFmpeg functions

• av_register_all(): Register all components, 4.0 has been deprecated

• avdevice_register_all() registers devices, such as V4L2, etc.

• avformat_network_init(); Initialize network libraries and libraries related to network encryption protocols (such as openssl)

8. Introduction to FFmpeg functions - packaging format related

• avformat_alloc_context();Responsible for applying for the memory of an AVFormatContext structure and performing simple initialization

• avformat_free_context(); Release everything in the structure and the structure itself

• avformat_close_input(); Close the demultiplexer. After closing, there is no need to use avformat_free_context to free it.

• avformat_open_input(); Open the input video file

• avformat_find_stream_info(): Get audio and video file information

• av_read_frame(); read audio and video packets

• avformat_seek_file(); locate the file

• av_seek_frame(): locate the file

9. Introduction to FFmpeg decoding function - decoder related

• avcodec_alloc_context3(): allocate decoder context

• avcodec_find_decoder(): Find decoder by ID

• avcodec_find_decoder_by_name(): According to the decoder name

• avcodec_open2(): Open codec

• avcodec_decode_video2(): decode a frame of video data

• avcodec_decode_audio4(): decode a frame of audio data

• avcodec_send_packet(): send encoded data packet

• avcodec_receive_frame(): receive decoded data

• avcodec_free_context(): Release the decoder context, including avcodec_close()

• avcodec_close(): Close the decoder

10. FFmpeg component registration method

10.1 FFmpeg 3.x component registration method

When we use ffmpeg, we must first execute av_register_all to register the global decoder, encoder and other structures into their respective global object linked lists, so that they can be searched and called later.

10.2 FFmpeg 4.x component registration method

FFmpeg does it internally, and does not require users to call the API to register. Take the codec codec as an example:

1. Generate components to be registered during configure./configure:7203:print_enabled_components libavcodec/codec_list.c AVCodec codec_list $CODEC_LIST A codec_list.c file will be generated here, which contains only static const AVCodec * const codec_list[] array.

2. In libavcodec/allcodecs.c, the codecs of static const AVCodec * const codec_list[] are organized in a linked list.

10.3 FFmpeg 4.0.2 component registration method

FFmepg does it internally, and does not require users to call the API to register.

For demuxer/muxer (demultiplexer, also known as container), it corresponds to

1. libavformat/muxer_list.c libavformat/demuxer_list.c These two files are also generated during configure, that is to say, there are no these two files when directly downloading the source code.

2. In libavformat/allformats.c, demuxer_list[] and muexr_list[] are organized in a linked list.

Other components are similar

[Learning Address]: Advanced Development of FFmpeg/WebRTC/RTMP/NDK/Android Audio and Video Streaming Media
[Article Benefits]: Receive more audio and video learning materials packages, Dachang interview questions, technical videos and learning roadmaps for free, including ( C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) If you need it, you can click 1079654574 to join the group to receive it~

11. Introduction to FFmpeg data structure

The AVFormatContext encapsulation format context structure is also a global structure that stores information about the video file encapsulation format.

AVInputFormat demuxer Each package format (such as FLV, MKV, MP4, AVI) corresponds to a structure.

AVOutputFormat muxer

Each video (audio) stream in the AVStream video file corresponds to this structure.

AVCodecContext codec context structure, which saves video (audio) codec related information.

AVCodec Each video (audio) codec (eg H.264 decoder) corresponds to this structure.

AVPacket stores a frame of compressed encoded data.

AVFrame stores a frame of decoded pixel (sample) data.

If the context data is stored in the decoder? There must be conflicts in the data during multi-channel decoding.

12. The relationship between FFmpeg data structures

12.1 The relationship between AVFormatContext and AVInputFormat

AVFormatContext API call AVInputFormat is mainly FFMPEG internal call

12.2 The relationship between AVCodecContext and AVCodec

AVCodecContext encoder context structure struct AVCodec *codec;

AVCodec Each video (audio) codec int (*decode)(AVCodecContext *, void *outdata, int *outdata_size, AVPacket avpkt); int ( encode2)( AVCodecContext *avctx, AVPacket *avpkt, const AVFrame *frame, int *got_packet_ptr);

12.3 The relationship between AVFormatContext, AVStream and AVCodecContext

12.4 Differentiate between different code streams

• AVMEDIA_TYPE_VIDEO video stream

video_index = av_find_best_stream(ic, AVMEDIA_TYPE_VIDEO,-1,-1, NULL, 0)

• AVMEDIA_TYPE_AUDIO audio stream

audio_index = av_find_best_stream(ic, AVMEDIA_TYPE_AUDIO,-1,-1, NULL, 0)

There is also an index field in AVPacket

14. FFmpeg data structure analysis

AVFormatContext

• iformat: AVInputFormat of input media, such as pointing to AVInputFormat ff_flv_demuxer

• nb_streams: AVStream number of input media

• streams: AVStream[] array of input media

• duration: the duration of the input media (in microseconds), the calculation method can refer to the av_dump_format() function.

• bit_rate: the bit rate of the input media

AVInputFormat

• name: encapsulation format name

• extensions: the extension of the package format

• id: encapsulation format ID

• Some interface functions for package format processing, such as read_packet()

AVStream

• index: identifies the video/audio stream

• time_base: the time base of the stream, PTS*time_base=real time (seconds)

• avg_frame_rate: the frame rate of the stream

• duration: the video/audio stream length

• codecpar: codec parameter attribute

AVCodecParameters

• codec_type: media type, such as AVMEDIA_TYPE_VIDEO AVMEDIA_TYPE_AUDIO, etc.

• codec_id: codec type, such as AV_CODEC_ID_H264 AV_CODEC_ID_AAC, etc.

AVCodecContext

• codec: AVCodec of the codec, such as pointing to AVCodec ff_aac_latm_decoder

• width, height: the width and height of the image (only for video)

• pix_fmt: pixel format (only for video)

• sample_rate: sample rate (only for audio)

• channels: number of channels (only for audio)

• sample_fmt: sample format (only for audio)

AVCodec

• name: codec name

• type: codec type

• id: Codec ID

• Some codec interface functions, such as int (*decode)()

AVCodecContext

• codec: AVCodec of the codec, such as pointing to AVCodec ff_aac_latm_decoder

• width, height: the width and height of the image (only for video)

• pix_fmt: pixel format (only for video)

• sample_rate: sample rate (only for audio)

• channels: number of channels (only for audio)

• sample_fmt: sample format (only for audio)

AVCodec

• name: codec name

• type: codec type

• id: Codec ID

• Some codec interface functions, such as int (*decode)()

The AVPacket

• pts: display timestamp

• dts: decoded timestamp

• data: compressed encoded data

• size: Compression encoded data size

• pos: Offset address of data

• stream_index: belonging AVStream

AVFrame

• data: decoded image pixel data (audio sample data)

• linesize: for video, the size of a line of pixels in the image; for audio, the size of the entire audio frame

• width, height: the width and height of the image (only for video)

• key_frame: whether it is a key frame (only for video).

• pict_type: frame type (only for video). e.g. I, P, B

• sample_rate: audio sample rate (only for audio)

• nb_samples: audio samples per channel (only for audio)

• pts: display time

15. FFmpeg memory model

◼ When copying a new Packet from an existing Packet, there are two situations:

• ① The bufs of the two Packets refer to the same data cache space. At this time, attention should be paid to the release of the data cache space;

• ② The bufs of the two Packets refer to different data cache spaces, and each Packet has a copy of the data cache space;

• For multiple AVPackets sharing the same cache space, the reference counting mechanism (reference-count) used by FFmpeg:

  • Initialize the reference count to 0, and only when AVBuffer is actually allocated, the reference count is initialized to 1;

  • When a new Packet references the shared cache space, the reference count is +1;

  • When the Packet that refers to the shared space is released, the reference count is -1; when the reference count is 0, the referenced cache space AVBuffer is released.

• AVFrame also uses the same mechanism.

16. Common APIs of AVPacket

• AVPacket *av_packet_alloc(void); Allocating AVPacket has nothing to do with buffer at this time

• void av_packet_free(AVPacket **pkt); Release AVPacket corresponding to _alloc

• void av_init_packet(AVPacket *pkt); Initializing AVPacket is simply initializing the pkt field

• int av_new_packet(AVPacket *pkt, int size); Allocate memory for AVPacket's buf, and initialize the reference count to 1

• int av_packet_ref(AVPacket *dst, const AVPacket *src); increase reference count

• void av_packet_unref(AVPacket *pkt); reduce reference count

• void av_packet_move_ref(AVPacket *dst, AVPacket *src); transfer reference count

• AVPacket *av_packet_clone(const AVPacket *src); equal to_packet_alloc()+of_packet_ref()

• AVFrame *av_frame_alloc(void); assignAVFrame

• void av_frame_free(AVFrame **frame); releaseAVFrame

• int av_frame_ref(AVFrame *dst, const AVFrame *src); increase reference count

• void av_frame_unref(AVFrame *frame); Decrease the reference count

• void av_frame_move_ref(AVFrame *dst, AVFrame *src); transfer reference count

• int av_frame_get_buffer(AVFrame *frame, int align); allocate memory according to AVFrame

• AVFrame *av_frame_clone(const AVFrame *src); equalsav_frame_alloc()+av_frame_ref()

Original link: Audio and video stereotyped essay (6) -- ffmpeg general introduction and memory model - Nuggets