(Audio and video study notes): FFmpeg memory model and common APIs of AVPacket and AVFrame

• [Description] Course learning address: https://ke.qq.com/course/468797

table of Contents

FFmpeg memory model

FFmpeg memory model-reference counting

AVPacket common API

AVFrame common API

API application introduction

FFmpeg memory model

• After reading an AVPacket from av_read_frame, how to put it into the decoder queue?
• After reading an AVFrame from avcodec_recevice_frame, how to put it into the decompressed frame queue?
   
• When copying a new Packet from an existing Packet, there are two situations :
  • ① The bufs of the two Packets refer to the same data cache space . At this time, attention should be paid to the release of the data cache space.
  • ② Packet of two references do not buf different data buffer space , and each has a copy Packet data cache space.

• Memory model

• A more accurate model

FFmpeg memory model-reference counting

• For multiple AVPackets sharing the same cache space, the reference-count mechanism (reference-count) used by FFmpeg :
  • The reference count is initialized to 0, only when the AVBuffer is actually allocated, the reference count is initialized to 1
  • When a new Packet references the shared cache space, the reference count is increased by +1
  • When the Packet that references the shared space is released, the reference count is -1
  • When the reference count is 0, the referenced buffer space AVBuffer is released
• AVFrame also uses the same mechanism.

AVPacket common API

AVPacket * av_packet _alloc (void);	Allocating AVPacket has nothing to do with buffer at this time
void av_packet _free (AVPacket ** pkt);	Release AVPacket and _alloc correspondence
void av_init_packet (AVPacket * pkt);	Initializing AVPacket is simply initializing the pkt field
int av_new_packet (AVPacket * pkt, int size);	Allocate memory to the buf of AVPacket, and the reference count is initialized to 1
int av_packet_ ref (AVPacket * dst, const AVPacket * src);	Increase the reference count
void av_packet_ unref (AVPacket * pkt);	Decrease the reference count
void av_packet_move_ref(AVPacket *dst, AVPacket *src);	Transfer reference count
AVPacket * av_packet_clone (const AVPacket * src);	等于 av_packet_alloc () + av_packet_ref (

AVFrame common API

AVFrame * av_frame_ alloc (void);	Assign AVFrame
void av_frame_free(AVFrame **frame);	Release AVFrame
int av_frame_ref(AVFrame *dst, const AVFrame *src);	Increase the reference count
void av_frame_unref(AVFrame *frame);	Decrease the reference count
void av_frame_move_ref(AVFrame *dst, AVFrame *src);	Transfer reference count
int av_frame_get_buffer(AVFrame *frame, int align);	Allocate memory according to AVFrame
AVFrame *av_frame_clone(const AVFrame *src);	等于 av_frame_alloc () + av_frame_ref ()

API application introduction

• av_packet_alloc simply creates an AVPacket and sets its field to the default value (data is empty and there is no data buffer space), and the data pointer needs to be assigned another value .

AVPacket *av_packet_alloc(void);

int av_new_packet(AVPacket *pkt, int size);

• Release the AVPacket created with  av_packet_alloc  . If the packet has a reference count (packet->buf is not empty), call av_packet_unref(&packet) first .
  • Only when the reference count is 0 , the data cache will be released when av_packet_free() is called .

void av_packet_free(AVPacket **pkt);

• The av_init_packet initializes the packet value to the default value . This function does not affect the data buffer space and size referenced by the buffer , and needs to be processed separately.
  • The buf in AVPacket is empty.
  • For example, call av_init_packet in av_get_packet

void av_init_packet(AVPacket *pkt);

• av_packet_ref uses a shallow copy of the reference count
  • This function will first copy all non-cached data, and then create a new reference count of src->buf.
  • If src has already set the reference count (src->buf is not empty), then directly add its reference count to 1.
  • If src does not set the reference count (src->buf is empty), create a new reference count buf for dst, and copy src->data to buf->buffer.
  • Finally, copy the other fields of src to dst. So av_packet_ref() uses two AVPackets to share a cache.

int av_packet_ref(AVPacket *dst, const AVPacket *src);

• av_packet_unref uses reference counting to clean up data
  • Set the reference count of the cache space to -1, and set other fields in the Packet to initial values.
  • If the reference count is 0, the cache space is automatically released .

void av_packet_unref(AVPacket *pkt);

• av_packet_move_ref  directly assigns the entire structure of src to dst, so the reference count does not change, and src is reset by av_init_packet
  • Its function is av_packet_alloc() + av_packet_ref();

void av_packet_move_ref(AVPacket *dst, AVPacket *src);

• av_packet_clone first creates a new AVPacket, and then performs counting reference + data copy, so that the new AVPacket points to the same data of the old AVPacket.

AVPacket *av_packet_clone(const AVPacket *src);

• AVPacket case:

/**
 * @brief 测试av_packet_alloc和av_packet_free的配对使用
 */
void av_packet_test1()
{
    AVPacket *pkt = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE); // 引用计数初始化为1
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);
    av_packet_unref(pkt);       // 要不要调用
    av_packet_free(&pkt);       // 如果不free将会发生内存泄漏,内部调用了 av_packet_unref
}

/**
 * @brief 测试误用av_init_packet将会导致内存泄漏
 */
void av_packet_test2()
{
    AVPacket *pkt = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);
//    av_init_packet(pkt);        // 这个时候init就会导致内存无法释放
    av_packet_free(&pkt);
}

/**
 * @brief 测试av_packet_move_ref后，可以av_init_packet
 */
void av_packet_test3()
{
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);
    pkt2 = av_packet_alloc();   // 必须先alloc
    av_packet_move_ref(pkt2, pkt);//内部其实也调用了av_init_packet
    av_init_packet(pkt);
    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}
/**
 * @brief 测试av_packet_clone
 */
void av_packet_test4()
{
    AVPacket *pkt = NULL;
    // av_packet_alloc()没有必要，因为av_packet_clone内部有调用 av_packet_alloc
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);
    pkt2 = av_packet_clone(pkt); // av_packet_alloc()+av_packet_ref()
    av_init_packet(pkt);
    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

/**
 * @brief 测试av_packet_ref
 */
void av_packet_test5()
{
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc(); //
    if(pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {    printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }

    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    if(pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {    printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);

    pkt2 = av_packet_alloc();   // 必须先alloc
    av_packet_move_ref(pkt2, pkt); // av_packet_move_ref
//    av_init_packet(pkt);  //av_packet_move_ref

    av_packet_ref(pkt, pkt2);
    av_packet_ref(pkt, pkt2);     // 多次ref如果没有对应多次unref将会内存泄漏
    if(pkt->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {    printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt->buf));
    }
    if(pkt2->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {    printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt2->buf));
    }
    av_packet_unref(pkt);   // 将为2
    av_packet_unref(pkt);   // 做第二次是没有用的
    if(pkt->buf)
        printf("pkt->buf没有被置NULL\n");
    else
        printf("pkt->buf已经被置NULL\n");
    if(pkt2->buf)        // 打印referenc-counted，必须保证传入的是有效指针
    {    printf("%s(%d) ref_count(pkt) = %d\n", __FUNCTION__, __LINE__,
               av_buffer_get_ref_count(pkt2->buf));
    }
    av_packet_unref(pkt2);


    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

/**
 * @brief 测试AVPacket整个结构体赋值, 和av_packet_move_ref类似
 */
void av_packet_test6()
{
    AVPacket *pkt = NULL;
    AVPacket *pkt2 = NULL;
    int ret = 0;

    pkt = av_packet_alloc();
    ret = av_new_packet(pkt, MEM_ITEM_SIZE);
    memccpy(pkt->data, (void *)&av_packet_test1, 1, MEM_ITEM_SIZE);

    pkt2 = av_packet_alloc();   // 必须先alloc
    *pkt2 = *pkt;   // 有点类似  pkt可以重新分配内存
    av_init_packet(pkt);

    av_packet_free(&pkt);
    av_packet_free(&pkt2);
}

• AVFrame case test

void av_frame_test()
{
    AVFrame *frame = NULL;
    int ret = 0;

    frame = av_frame_alloc();// 没有类似的AVPacket的av_new_packet的API
    // 1024 *2 * (16/8) =
    frame->nb_samples     = 1024;
    frame->format         = AV_SAMPLE_FMT_S16;//AV_SAMPLE_FMT_S16P AV_SAMPLE_FMT_S16
    frame->channel_layout = AV_CH_LAYOUT_STEREO;    //AV_CH_LAYOUT_MONO AV_CH_LAYOUT_STEREO
    ret = av_frame_get_buffer(frame, 0);    // 根据格式分配内存
    if(frame->buf && frame->buf[0])
        printf("%s(%d) 1 frame->buf[0]->size = %d\n", __FUNCTION__, __LINE__, frame->buf[0]->size);    //受frame->format等参数影响
    if(frame->buf && frame->buf[1])
        printf("%s(%d) 1 frame->buf[1]->size = %d\n", __FUNCTION__, __LINE__, frame->buf[1]->size);    //受frame->format等参数影响

    if(frame->buf && frame->buf[0])        // 打印referenc-counted，必须保证传入的是有效指针
        printf("%s(%d) ref_count1(frame) = %d\n", __FUNCTION__, __LINE__, av_buffer_get_ref_count(frame->buf[0]));

    ret = av_frame_make_writable(frame);    // 当frame本身为空时不能make writable
    printf("av_frame_make_writable ret = %d\n", ret);

    if(frame->buf && frame->buf[0])        // 打印referenc-counted，必须保证传入的是有效指针
        printf("%s(%d) ref_count2(frame) = %d\n", __FUNCTION__, __LINE__, av_buffer_get_ref_count(frame->buf[0]));

    av_frame_unref(frame);
    if(frame->buf && frame->buf[0])        // 打印referenc-counted，必须保证传入的是有效指针
        printf("%s(%d) ref_count3(frame) = %d\n", __FUNCTION__, __LINE__, av_buffer_get_ref_count(frame->buf[0]));

    av_frame_free(&frame);
}