Overall overview of ijkplayer source code analysis

Preface

This series is a source code analysis of ijkPlayer. This article is an overall overview of ijkPlayer, analyzing the main process, main structure, and initialization process to lay the foundation for the following articles.

• This series is as follows:
  Overall overview
  video rendering process
  audio playback process
  read thread process
  audio decoding process
  video decoding process
  video to audio synchronization

Main process

read_thread -> PacketQueue(AVPacket) -> FrameQueue(AVFrame) -> 渲染

Correspondence	PacketQueue	FrameQueue	Clock	Decoding thread	Rendering thread
video	videoq	pictq	vidclk	video_thread	video_refresh_thread
Audio	audioq	sampq	audclk	audio_thread	aout_thread
subtitle	subtitleq	subpq	extclk	subtitle_thread	video_refresh_thread

• Structure diagram
  

• flow chart
  

Main structure

IjkMediaPlayer, FFPlayer, IJKFF_Pipeline are created in the native_setup method, and
VideoState is created through stream_open in the prepare phase.

IjkMediaPlayer

The Player representing the native layer is bound one-to-one with the Java layer. As an entry package from Java to C.

struct IjkMediaPlayer {
    
    
    volatile int ref_count;
    pthread_mutex_t mutex;
    FFPlayer *ffplayer;

    int (*msg_loop)(void*);
    SDL_Thread *msg_thread;
    SDL_Thread _msg_thread;

    int mp_state;
    char *data_source;
    void *weak_thiz;

    int restart;
    int restart_from_beginning;
    int seek_req;
};

FFPlayer

The specific player, internal player, includes encoding, output, etc.; after the code enters ff_ffplay.c, FFPlayer is used. Holds VideoState.

typedef struct FFPlayer {
    
    
    VideoState *is;

    /* extra fields */
    SDL_Aout *aout;
    SDL_Vout *vout;
    struct IJKFF_Pipeline *pipeline;
    struct IJKFF_Pipenode *node_vdec;

    MessageQueue msg_queue;
}

VideoState

VideoState, created in stream_open, represents all states during playback.

typedef struct VideoState {
    
    
    Clock audclk;
    Clock vidclk;
    Clock extclk;

    FrameQueue pictq;
    FrameQueue subpq;
    FrameQueue sampq;

    Decoder auddec;
    Decoder viddec;
    Decoder subdec;

    PacketQueue audioq;
    PacketQueue subtitleq;
    PacketQueue videoq;

    int seek_req;
    double frame_timer;
    int abort_request;
    int force_refresh;
    int paused;

}

IJKFF_Pipeline

ffpipeline encapsulates the video decoder and audio decoder/output, but the implementation is to call others through function pointers, a bit like facade mode.

Defined in ff_ffpipeline.h and ff_ffpipeline.c

// ffpipeline_android.c中
typedef struct IJKFF_Pipeline_Opaque {
    
    
    FFPlayer      *ffp;
    SDL_mutex     *surface_mutex;
    jobject        jsurface;
    volatile bool  is_surface_need_reconfigure;

    bool         (*mediacodec_select_callback)(void *opaque, ijkmp_mediacodecinfo_context *mcc);
    void          *mediacodec_select_callback_opaque;

    SDL_Vout      *weak_vout;

    float          left_volume;
    float          right_volume;
} IJKFF_Pipeline_Opaque;

// 在ff_ffpipeline.h
typedef struct IJKFF_Pipeline_Opaque IJKFF_Pipeline_Opaque;
typedef struct IJKFF_Pipeline IJKFF_Pipeline;
struct IJKFF_Pipeline {
    
    
    SDL_Class             *opaque_class;
    IJKFF_Pipeline_Opaque *opaque;

    void            (*func_destroy)             (IJKFF_Pipeline *pipeline);
    IJKFF_Pipenode *(*func_open_video_decoder)  (IJKFF_Pipeline *pipeline, FFPlayer *ffp);
    SDL_Aout       *(*func_open_audio_output)   (IJKFF_Pipeline *pipeline, FFPlayer *ffp);
    IJKFF_Pipenode *(*func_init_video_decoder)  (IJKFF_Pipeline *pipeline, FFPlayer *ffp);
    int           (*func_config_video_decoder)  (IJKFF_Pipeline *pipeline, FFPlayer *ffp);
};

IJKFF_Pipeline *ffpipeline_alloc(SDL_Class *opaque_class, size_t opaque_size);
void ffpipeline_free(IJKFF_Pipeline *pipeline);
void ffpipeline_free_p(IJKFF_Pipeline **pipeline);

// 打开视频解码器
IJKFF_Pipenode *ffpipeline_open_video_decoder(IJKFF_Pipeline *pipeline, FFPlayer *ffp);
// 打开音频解码器
SDL_Aout       *ffpipeline_open_audio_output(IJKFF_Pipeline *pipeline, FFPlayer *ffp);

// 异步初始化IJKFF_Pipenode
IJKFF_Pipenode* ffpipeline_init_video_decoder(IJKFF_Pipeline *pipeline, FFPlayer *ffp); 

// 异步初始化视频解码器
int ffpipeline_config_video_decoder(IJKFF_Pipeline *pipeline, FFPlayer *ffp);

serial field

The serial field is mainly used to mark the serial number of the current node. The concept of serial is used in many places in ffplay, which is generally used to distinguish continuous data. When seek, it will trigger the serial field change.
In general, the serial of the newly added node is the same as the serial of the previous node, but when a flush_pkt is added to the queue, the serial of the subsequent node will be 1 larger than the previous one.
Serial is the value in Packet, PacketQueue, Frame, and Clock, and it will be judged in various places.
There is pkt_serial in Decoder, which is taken from pkt every time a Packet is decoded.

Initialization process

native_setup process

• new ijkMediaPlayer, the constructor calls native_setup
  ->ijkplayer_jni.c#IjkMediaPlayer_native_setup
  ->ijkplayer_android.c#ijkmp_android_create: Create IjkMediaPlayer, FFPlayer, IJKFF_Pipeline, etc.

// ijkplayer_android.c
IjkMediaPlayer *ijkmp_android_create(int(*msg_loop)(void*))
{
    
    
    // 创建IjkMediaPlayer、FFPlayer
    IjkMediaPlayer *mp = ijkmp_create(msg_loop);  

    //  创建SDL_Vout
    mp->ffplayer->vout = SDL_VoutAndroid_CreateForAndroidSurface();

    // 创建IJKFF_Pipeline
    mp->ffplayer->pipeline = ffpipeline_create_from_android(mp->ffplayer);

    // 互相绑定
    ffpipeline_set_vout(mp->ffplayer->pipeline, mp->ffplayer->vout);

    return mp;
}

IjkMediaPlayer *ijkmp_create(int (*msg_loop)(void*))
{
    
    
    IjkMediaPlayer *mp = (IjkMediaPlayer *) mallocz(sizeof(IjkMediaPlayer));
    mp->ffplayer = ffp_create();
    mp->msg_loop = msg_loop; // 赋值，后续prepare会启动线程调用该方法
    ijkmp_inc_ref(mp);
    pthread_mutex_init(&mp->mutex, NULL);

    return mp;
}

FFPlayer *ffp_create() {
    
    
    FFPlayer *ffp = (FFPlayer *) av_mallocz(sizeof(FFPlayer));

    msg_queue_init(&ffp->msg_queue);
    ffp->af_mutex = SDL_CreateMutex();
    ffp->vf_mutex = SDL_CreateMutex();
    ffp->stat_mutex = SDL_CreateMutex();

    ffp_reset_internal(ffp);
    ffp->av_class = &ffp_context_class;
    ffp->meta = ijkmeta_create();

    av_opt_set_defaults(ffp);

    return ffp;
}

prepare process

• VideoState
  is created in stream_open, encapsulating all the parameters required in the process, a hodgepodge.

• Java layer prepareAsync
  -> ijkplayer_jni.c#IjkMediaPlayer_prepareAsync
  -> ijkplayer.c#ijkmp_prepare_async
  -> ijkplayer.c#ijkmp_prepare_async_l, create ijkmp_msg_loopthread, start msg_queue
  -> ff_ffplay.c#ffp_prepare_async_out, play and create SDLopen_out

static int ijkmp_prepare_async_l(IjkMediaPlayer *mp){
    
    
    ijkmp_change_state_l(mp, MP_STATE_ASYNC_PREPARING); // 改变状态
    msg_queue_start(&mp->ffplayer->msg_queue); // 启动message_queue
    SDL_CreateThreadEx(&mp->_msg_thread, ijkmp_msg_loop, mp, "ff_msg_loop");
    ffp_prepare_async_l(mp->ffplayer, mp->data_source)
}

int ffp_prepare_async_l(FFPlayer *ffp, const char *file_name){
    
    
    VideoState *is = stream_open(ffp, file_name, NULL);
    ffp->is = is;
}

Other points

• so load, call JNI_OnLoad method, perform related initialization operations
  libLoader.loadLibrary("ijkffmpeg"); // ffmpeg
  libLoader.loadLibrary("ijksdl"); // sdl
  libLoader.loadLibrary("ijkplayer"); // player, file dependency Ijkj4a, only these three so

• Java layer call:
  new ijkMediaPlayer, the constructor calls native_setup
  mMediaPlayer.setDataSource
  mMediaPlayer.setDisplay
  mMediaPlayer.prepareAsync
  mMediaPlayer.start

• The message queue will be analyzed separately later in
  ijkplayer_jni.c# static int message_loop(void *arg)
  ijkplayer.c#prepare to create a thread and start the loop; the
  message queue is the set of looper implemented using the lock mechanism, waiting to wake up the implemented looper.

• Code structure
  

Refer to
ijkPlayer main process analysis
ijkplayer framework in-depth analysis and
analysis of IJKPlayer
with problems reread ijkPlayer
knows the column: audio and video technology