ijkplayer source code analysis (2): message distribution processing mechanism

1 Introduction

The previous blog ijkplayer source code analysis (1): Initialization process The 4.1.1 ijkmp_create() part briefly explained the message processing mechanism of ijkplayer. This article analyzes the source code in detail to find out its message mechanism and processing flow.

The player is a relatively complex multi-thread project, such as data reading thread, audio decoding thread, video decoding thread, video playback thread, etc. Each thread needs to generate events, such as state changes, errors, etc., which need to be passed to the control layer and business layer for processing. Therefore, a message processing mechanism is needed to complete these tasks, and convert the events generated by each thread into messages for processing. Of course, it can also be handled through interface callbacks, but events with various styles will require extremely large and complex interfaces, which will be difficult to maintain and expand. Therefore, the message mechanism is a very good solution.

By analyzing the source code of ijkplayer, you will find that the message processing mechanism implemented by ijkplayer is very similar to the message distribution and processing mechanism of Handler, Looper, Message, and MessageQueue on the Android side. Next, we will analyze the message distribution processing mechanism of ijkplayer in detail.

This article is based on the A4ijkplayer project for ijkplayer source code analysis. This project is to change ijkplayer to CMake-based compilation, which can be imported into Android Studio to compile and run, which is convenient for code search, function jump, single-step debugging, call stack trace, etc.

2. Message distribution processing mechanism

2.1 Android message distribution processing mechanism

Let's first review the message distribution and processing mechanism of Handler, Looper, Message, and MessageQueue on the Android side, which helps us understand the message processing mechanism of ijkplayer.

Android's message distribution processing mechanism is as follows:

• Send a message through the Handler's sendMessage() method

• Looper adds the message to the message queue MessageQueue

• The thread where the Looper is located loops through the MessageQueue to take out the message distribution, if there is no message, it hangs and waits

• Distribute to the Handler and execute handleMessage() to process the message. The message sending and processing process is completed above.

  

2.2 Message distribution processing mechanism of ijkplayer

As shown above, ijkplayer's message distribution processing mechanism is similar to that of Android, using the producer-consumer model. Producers (can be multiple) produce messages from any thread and add them to the message queue. Consumers (only one) take out messages from the message queue in an independent thread loop for distribution processing, and wait if the queue is empty. The message structure in ijkplayer is AVMessage, and the message queue structure is MessageQueue, both of which are located in ijkmedia/ijkplayer/ff_ffmsg_queue.hthe file . The structure is as follows, and it can be seen that it is very similar to Android's Message.

typedef struct AVMessage {
    int what;
    int arg1;
    int arg2;
    void *obj;
    void (*free_l)(void *obj);
    struct AVMessage *next;
} AVMessage;
​
typedef struct MessageQueue {
    AVMessage *first_msg, *last_msg;
    int nb_messages;
    int abort_request;
    SDL_mutex *mutex;
    SDL_cond *cond;
​
    AVMessage *recycle_msg;
    int recycle_count;
    int alloc_count;
} MessageQueue;

2.2.1 Message-related processing functions provided by Message:

// 初始化消息体，实际就是调用 memeset 重置内存
inline static void msg_init_msg(AVMessage *msg)
​
// 释放 msg 的参数 obj 内存，并将其置空，注：不是释放 msg 内存
inline static void msg_free_res(AVMessage *msg)

2.2.2 Message queue-related processing functions provided by MessageQueue:

// 初始化 MessageQueue
inline static void msg_queue_init(MessageQueue *q)
// 释放 MessageQueue 内所有 Message 内存及内部的 Mutex、Condition 内存，并不释放 MessageQueue 本身内存，需外部 free 或 delete
inline static void msg_queue_destroy(MessageQueue *q)
​
// 启用 MessageQueue 并填入一个 FFP_MSG_FLUSH 消息
inline static void msg_queue_start(MessageQueue *q)
// 终止使用 MessageQueue，调用后可通过 msg_queue_start 重新启用
inline static void msg_queue_abort(MessageQueue *q)
​
// 以下 put 系列是向 MessageQueue 添加消息，只是参数不同
inline static int msg_queue_put(MessageQueue *q, AVMessage *msg)
inline static void msg_queue_put_simple1(MessageQueue *q, int what)
inline static void msg_queue_put_simple2(MessageQueue *q, int what, int arg1)
inline static void msg_queue_put_simple3(MessageQueue *q, int what, int arg1, int arg2)
inline static void msg_queue_put_simple4(MessageQueue *q, int what, int arg1, int arg2, void *obj, int obj_len)
​
// 从队列队首取出消息
inline static int msg_queue_get(MessageQueue *q, AVMessage *msg, int block)
​
// 移除指定 msg.what 的消息
inline static void msg_queue_remove(MessageQueue *q, int what)
​
// 清空 MessageQueue
inline static void msg_queue_flush(MessageQueue *q)

2.2.3 How to create and destroy MessageQueue

 // 创建、初始化和启用 MessageQueue
MessageQueue* q = (MessageQueue*)malloc(sizeof(MessageQueue);
msg_queue_init(q);
msg_queue_start(q);
​
// put 消息
// get 消息
​
// 终止、销毁和释放 MessageQueue，创建时的逆过程
msg_queue_abort(q);
msg_queue_destroy(q);
free(q);
q = NULL;

3. Detailed explanation of ijkplayer message mechanism and process

Follow up and analyze the ijkplayer source code, let's sort out the creation process of the ijkplayer message mechanism. As shown below:

3.1 IjkMediaPlayer_native_setup()

According to the previous blog ijkplayer source code analysis (1): initialization process , we can see that the message mechanism is initially created when it is called in the Java layer constructor native_setup()and then called to the native layer.IjkMediaPlayer_native_setup()

static void
IjkMediaPlayer_native_setup(JNIEnv *env, jobject thiz, jobject weak_this)
{
    IjkMediaPlayer *mp = ijkmp_android_create(message_loop);
    // ...
}

Call to ijkmp_create()pass in the function pointer message_loop, and the content of the message_loop function will be analyzed later.

IjkMediaPlayer *ijkmp_android_create(int(*msg_loop)(void*))
{
    IjkMediaPlayer *mp = ijkmp_create(msg_loop);
    // ...
}

Then look at ijkmp_create()the function , which is called here ffp_create()and saves the msg_loop function pointer. Later, when preparingAsync, go to the msg_loop function through the IjkMediaPlayer area and put it into the message processing thread for execution.

IjkMediaPlayer *ijkmp_create(int (*msg_loop)(void*))
{
    IjkMediaPlayer *mp = (IjkMediaPlayer *) mallocz(sizeof(IjkMediaPlayer));
    // ...
    mp->ffplayer = ffp_create(); // 创建 FFPlayer
    // ...
    mp->msg_loop = msg_loop; // 保存 msg_loop 函数指针
}

Then look at ffp_create()the function , related to the message is to initialize the message queue and the status queue.

FFPlayer *ffp_create()
{
    // ...
    FFPlayer* ffp = (FFPlayer*) av_mallocz(sizeof(FFPlayer));
    // ...
    msg_queue_init(&ffp->msg_queue); // 初始化消息队列
    // ...
    ffp_reset_internal(ffp); // 内部会执行 msg_queue_flush() 清空消息队列
    return ffp;
}

The above native_setup()function created and initialized MessageQueue, and saved the message_loop function pointer to IjkMediaPlayer. When did the message queue and message processing thread start and flow?

[Learning address]: FFmpeg/WebRTC/RTMP/NDK/Android audio and video streaming media advanced development

[Article Benefits]: Receive more audio and video learning packages, Dachang interview questions, technical videos and learning roadmaps for free. The materials include (C/C++, Linux, FFmpeg webRTC rtmp hls rtsp ffplay srs, etc.) Click 1079654574 to join the group to receive it~

3.2 IjkMediaPlayer_prepareAsync()

Analyze the source code of ijkplayer and find that it is enabled prepareAsync()in .

static void
IjkMediaPlayer_prepareAsync(JNIEnv *env, jobject thiz)
{
    // ...
    retval = ijkmp_prepare_async(mp);
    // ...
}
​
int ijkmp_prepare_async(IjkMediaPlayer *mp)
{
    ...
    int retval = ijkmp_prepare_async_l(mp);
    ...
}

ijkmp_prepare_async_l()In does two things:

• Enable the message queue Call to msg_queue_start()enable the message queue, and add a FFP_MSG_FLUSH message (the message queue will be emptied after calling prepareAsync() after switching resources).

• Create and start a message processing thread. The thread name is: ff_msg_loop, and the function body executed by the thread is: ijkmp_msg_loop(). Its interior is actually the msg_loop function pointer passed in and saved in native_setup()the function , that is, message_loop()the function, as follows:

static int ijkmp_msg_loop(void *arg)
{
    IjkMediaPlayer *mp = arg;
    int ret = mp->msg_loop(arg);
    return ret;
}
​
static int ijkmp_prepare_async_l(IjkMediaPlayer *mp)
{
    // ...
    // 启用消息队列
    msg_queue_start(&mp->ffplayer->msg_queue); 
    // ...
    // 创建消息队列处理线程
    mp->msg_thread = SDL_CreateThreadEx(&mp->_msg_thread, ijkmp_msg_loop, mp, "ff_msg_loop");
    //...
    return 0;
}
​
inline static void msg_queue_start(MessageQueue *q)
{
    SDL_LockMutex(q->mutex);
    // 启用消息队列
    q->abort_request = 0;
​
    // 添加一个 FFP_MSG_FLUSH 消息
    AVMessage msg;
    msg_init_msg(&msg);
    msg.what = FFP_MSG_FLUSH;
    msg_queue_put_private(q, &msg);
    SDL_UnlockMutex(q->mutex);
}

3.3 Consumption and distribution of ijkplayer messages

Let's look at message_loop()the function , in ijkmedia/ijkplayer/android/ijkplayer_jni.cthe file, as follows:

static int message_loop(void *arg)
{
    // ...
    message_loop_n(env, mp);
    // ...
}
​
static void message_loop_n(JNIEnv *env, IjkMediaPlayer *mp)
{
    jobject weak_thiz = (jobject) ijkmp_get_weak_thiz(mp);
    while (1) {
        AVMessage msg;
        // 从消息队列取消息
        int retval = ijkmp_get_msg(mp, &msg, 1);
        if (retval < 0)
            break;
        switch (msg.what) {
            // ...
            case FFP_MSG_FLUSH:
                // 发送消息到上层
                post_event(env, weak_thiz, MEDIA_NOP, 0, 0);
                break;
            case FFP_MSG_PREPARED:
                post_event(env, weak_thiz, MEDIA_PREPARED, 0, 0);
                break;
            // ...
        }
        // 释放消息的参数内存
        msg_free_res(&msg);
    }
}

There are two main functions here:

• ijkmp_get_msg()

• post_event()。

Let's first look at getting messages ijkmp_get_msg()from the message queue:

int ijkmp_get_msg(IjkMediaPlayer *mp, AVMessage *msg, int block)
{
    while (1) {
        int continue_wait_next_msg = 0;
        // 从消息队列的队首取一个消息
        int retval = msg_queue_get(&mp->ffplayer->msg_queue, msg, block);
        if (retval <= 0)
            return retval;
        switch (msg->what) {...}
        if (continue_wait_next_msg) {
            msg_free_res(msg);
            continue;
        }
        return retval;
    }
    return -1;
}

Let's look at post_event()the function : In the above message_loop_n()function, after getting the FFP_MSG_FLUSH message added at msg_queue_start()time (the first message after the message queue is enabled) post_event(), the function will be called to send the msg.what = MEDIA_NOP event:

case FFP_MSG_FLUSH:
     // 发送消息到上层
     post_event(env, weak_thiz, MEDIA_NOP, 0, 0);
     break;
​
inline static void post_event(JNIEnv *env, jobject weak_this, int what, int arg1, int arg2)
{
    J4AC_IjkMediaPlayer__postEventFromNative(env, weak_this, what, arg1, arg2, NULL);
}

J4AC_IjkMediaPlayer__postEventFromNativeis a macro defined in ijkmedia/ijkj4a/j4a/class/tv/danmaku/ijk/media/player/IjkMediaPlayer.hthe file , and actually calls the following function:

void J4AC_tv_danmaku_ijk_media_player_IjkMediaPlayer__postEventFromNative(JNIEnv *env, jobject weakThiz, jint what, jint arg1, jint arg2, jobject obj)
{
    (*env)->CallStaticVoidMethod(env, class_J4AC_tv_danmaku_ijk_media_player_IjkMediaPlayer.id, class_J4AC_tv_danmaku_ijk_media_player_IjkMediaPlayer.method_postEventFromNative, weakThiz, what, arg1, arg2, obj);
}
method_postEventFromNative 的相关定义是在 ijkmedia/ijkj4a/j4a/class/tv/danmaku/ijk/media/player/IjkMediaPlayer.c 文件中，如下：

    class_id = class_J4AC_tv_danmaku_ijk_media_player_IjkMediaPlayer.id;
    name     = "postEventFromNative";
    sign     = "(Ljava/lang/Object;IIILjava/lang/Object;)V";
    class_J4AC_tv_danmaku_ijk_media_player_IjkMediaPlayer.method_postEventFromNative = J4A_GetStaticMethodID__catchAll(env, class_id, name, sign);

It corresponds to the static function of the Java layer postEventFromNative(), that is, the underlying call will post_envent()eventually be called back to the Java layer postEventFromNative()function (in IjkMediaPlayer.java) through JNI.

@CalledByNative
private static void postEventFromNative(Object weakThiz, int what,
        int arg1, int arg2, Object obj) {
    // ...
    if (mp.mEventHandler != null) {
        Message m = mp.mEventHandler.obtainMessage(what, arg1, arg2, obj);
        mp.mEventHandler.sendMessage(m);
    }
}

Then it will be sent to EventHandler for its handleMessage()processing , here will receive the first message of msg.what = MEDIA_NOP after the native message queue is enabled, Java ignores this message:

private static class EventHandler extends Handler {
    // ...
    @Override
    public void handleMessage(Message msg) {
        // ...
        switch (msg.what) {
        case MEDIA_PREPARED:
            player.notifyOnPrepared();
            return;
        case MEDIA_NOP: // interface test message - ignore
            break;
        // ...
    }
}

The above is the whole message mechanism and enabling process of ijkplayer, from Java to native and then to Java.

Original Link: ijkplayer Source Code Analysis (2): Message Distribution Processing Mechanism - Nuggets