iOS ijkplayer hard solution H265 (hevc) 4k video problem solving

Table of contents

1. Solve the problem of no sound in mp2 audio format

2. Solve the problem of video freeze, audio and video out of sync (caused by hard solution failure)

1. Create a player

2. Create a decoder

3. Determine the problem and solve it

---

Continued from the previous article: Android, iOS ijkplayer compilation steps and related problem solving

Previous articleBased on the open source project of station B : https://github.com/bilibili/ijkplayer

After successfully compiling the iOS version of ijkplayer, I performed a h265 and 4K (3840x2160) stream playback test, and found that no matter how I try to configure the parameters such as soft and hard solutions and frame loss exposed by the upper layer

- (void)viewWillAppear:(BOOL)animated {
    ...

    // 尝试硬解 0 是软解 1是硬解
    // 这里提前剧透一下：这里的硬解设置没有生效，下面会详细解释
    IJKFFOptions *options = [IJKFFOptions optionsByDefault];
    [options setPlayerOptionIntValue:1 forKey:@"videotoolbox"];

    // 尝试通过增加丢帧数来让视频尽快追上音频
    [options setPlayerOptionIntValue:5 forKey:@"framedrop"];

    printf("---- zs log ----IJKMoviePlayerViewController prepareToPlay \n");
    [self.player prepareToPlay];
}

cannot be played normally, specifically as follows:

• aac format audio playback is normal
• mp2 format audio no sound
• Video playback is very laggy
• In the aac audio format, the audio is played at normal speed, and the video is played slower and slower due to the freeze, so the audio and video out of sync phenomenon becomes more and more obvious with the increase of the playback time

Therefore, we have to start from the source code to analyze and solve the problem:

1. Solve the problem of no sound in mp2 audio format

This problem is relatively easy to solve, because the source code supports this format, you only need to add it to the configuration:

The configuration file I use here is module-lite.sh. If you use module-lite-hevc.sh or others, just change the corresponding configuration file.

Step 1: Open ijkplayer-ios/config/module-lite.sh

Step 2: Add below the # ./configure --list-decoders group:

export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=mpga"
export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-decoder=mp2"

Step 3: Add below the # ./configure --list-muxers group:

export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-muxer=mpga"
export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-muxer=mp2"

Step 4: Add below the # ./configure --list-demuxers group:

export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-demuxer=mpga"
export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-demuxer=mp2"

Step 5: Add below the # ./configure --list-parsers group:

export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-parser=mpga"
export COMMON_FF_CFG_FLAGS="$COMMON_FF_CFG_FLAGS --enable-parser=mp2"

Last screenshot:

Step 6: Go to the config path and delete module.sh, then execute:

ln -s module-lite.sh module.sh

     Or go to the root directory and execute:

./init-config.sh（此步骤会自动拷贝module-lite.sh文件内容到module.sh）

To put it bluntly, you can also manually copy all the contents of module-lite.sh to module.sh without deleting module.sh

Step 7: Recompile after clean

./compile-ffmpeg.sh clean
./compile-ffmpeg.sh all

2. Solve the problem of video freeze, audio and video out of sync (caused by hard solution failure)

Note: If the mobile phone is below iPhone 7 or the iOS version is below 11.0, it does not support hardware solution.

This is the highlight of the analysis in this article. We locate and solve the problem step by step in the process of sorting out the core process of the player:

1. Create a player

Call the initWithContentURL method of IJKFFMoviePlayerControll from the upper layer, and pass in the playback address:

- (id)initWithContentURL:(NSURL *)aUrl withOptions:(IJKFFOptions *)options
{
...
    NSString *aUrlString = [aUrl isFileURL] ? [aUrl path] : [aUrl absoluteString];
    return [self initWithContentURLString:aUrlString withOptions:options];
}

Continue to call to: initWithContentURLString method

- (id)initWithContentURLString:(NSString *)aUrlString
                   withOptions:(IJKFFOptions *)options
{
    if (aUrlString == nil)
        return nil;

    self = [super init];
    if (self) {
        ijkmp_global_init();
        ijkmp_global_set_inject_callback(ijkff_inject_callback);

        [IJKFFMoviePlayerController checkIfFFmpegVersionMatch:NO];

        if (options == nil)
            options = [IJKFFOptions optionsByDefault];

        ...
        // init player --- 关键方法 初始化播放器
        _mediaPlayer = ijkmp_ios_create(media_player_msg_loop);
        ...
}

Initialize the player: _mediaPlayer = ijkmp_ios_create(media_player_msg_loop);

This line of code will be called into the ijkplayer_ios.m file

IjkMediaPlayer *ijkmp_ios_create(int (*msg_loop)(void*))
{
    //这是我们要关注的核心方法1
    IjkMediaPlayer *mp = ijkmp_create(msg_loop);
    if (!mp)
        goto fail;

    mp->ffplayer->vout = SDL_VoutIos_CreateForGLES2();
    if (!mp->ffplayer->vout)
        goto fail;

    //这是我们要关注的核心方法2
    mp->ffplayer->pipeline = ffpipeline_create_from_ios(mp->ffplayer);
    if (!mp->ffplayer->pipeline)
        goto fail;

    return mp;

fail:
    ijkmp_dec_ref_p(&mp);
    return NULL;
}

The ijkmp_ios_create method mainly does two things:

• The IjkMediaPlayer player is created by the ijkmp_create method
• Created ffpipeline through ffpipeline_create_from_ios (can be understood as a decoder and audio output provider)

The creation process is now complete.

2. Create a decoder

Or start with the call from the upper layer:

IJKFFMoviePlayerController.m  --- >  prepareToPlay

- (void)prepareToPlay
{
    //设置播放url
    ijkmp_set_data_source(_mediaPlayer, [_urlString UTF8String]);
    ...    
    //异步准备
    ijkmp_prepare_async(_mediaPlayer);
}

Follow up to: ijkplayer.c ---> ijkmp_prepare_async

int ijkmp_prepare_async(IjkMediaPlayer *mp)
{
    assert(mp);
    MPTRACE("ijkmp_prepare_async()\n");
    pthread_mutex_lock(&mp->mutex);
    //关键流程方法
    int retval = ijkmp_prepare_async_l(mp);
    pthread_mutex_unlock(&mp->mutex);
    MPTRACE("ijkmp_prepare_async()=%d\n", retval);
    return retval;
}

Continue to look at ijkplayer.c ---> ijkmp_prepare_async_l method

static int ijkmp_prepare_async_l(IjkMediaPlayer *mp)
{
...
    // released in msg_loop
    ijkmp_inc_ref(mp);
    ...
    //关键方法
    int retval = ffp_prepare_async_l(mp->ffplayer, mp->data_source);
    ...
    return 0;
}

Follow up to: ff_ffplay.c ---> ffp_prepare_async_l method

int ffp_prepare_async_l(FFPlayer *ffp, const char *file_name)
{
    ...
    //关键方法
    VideoState *is = stream_open(ffp, file_name, NULL);
}

Enter: ff_ffplay.c ---> stream_open method

static VideoState *stream_open(FFPlayer *ffp, const char *filename, AVInputFormat *iformat)
{
    ...
    //ZS:启动read_thread 线程，在线程中会根据读取的文件或者流的信息去判断是否存在音频流和视频流，然后通过 stream_component_open 方法找到对应的解码器，启动解码线程。
    is->read_tid = SDL_CreateThreadEx(&is->_read_tid, read_thread, ffp, "ff_read");
...
}

Enter ff_ffplay.c ---> read thread to see:

/* this thread gets the stream from the disk or the network */
//ZS:读取文件或网络流的信息
static int read_thread(void *arg)
{
    ...
    /* open the streams */
    if (st_index[AVMEDIA_TYPE_AUDIO] >= 0) {
        /* ZS:音频存在 去查找对应的解码器 启动解码线程*/
        stream_component_open(ffp, st_index[AVMEDIA_TYPE_AUDIO]);
    } else {
        ffp->av_sync_type = AV_SYNC_VIDEO_MASTER;
        is->av_sync_type  = ffp->av_sync_type;
    }
    ret = -1;
    if (st_index[AVMEDIA_TYPE_VIDEO] >= 0) {
        /* ZS:视频存在 去查找对应的解码器 启动解码线程*/
        ret = stream_component_open(ffp, st_index[AVMEDIA_TYPE_VIDEO]);
    }
    ...
}

Here we only focus on the video, enter ff_ffplay.c ---> stream_component_open method

/* open a given stream. Return 0 if OK */
static int stream_component_open(FFPlayer *ffp, int stream_index)
{
...
    case AVMEDIA_TYPE_VIDEO:
        
        if (ffp->async_init_decoder) {
            while (!is->initialized_decoder) {
                SDL_Delay(5);
            }
            if (ffp->node_vdec) {
                is->viddec.avctx = avctx;
                ret = ffpipeline_config_video_decoder(ffp->pipeline, ffp);
            }
            if (ret || !ffp->node_vdec) {
                decoder_init(&is->viddec, avctx, &is->videoq, is->continue_read_thread);
                ffp->node_vdec = ffpipeline_open_video_decoder(ffp->pipeline, ffp);
                if (!ffp->node_vdec)
                    goto fail;
            }
        } else {
            decoder_init(&is->viddec, avctx, &is->videoq, is->continue_read_thread);
            //打开视频解码器，我们这里是同步的情况
            ffp->node_vdec = ffpipeline_open_video_decoder(ffp->pipeline, ffp);
            if (!ffp->node_vdec)
                goto fail;
        }
        ...
}

Follow up to ff_ffpipeline.c --->ffpipeline_open_video_decoder

IJKFF_Pipenode* ffpipeline_open_video_decoder(IJKFF_Pipeline *pipeline, FFPlayer *ffp)
{
    return pipeline->func_open_video_decoder(pipeline, ffp);
}

Continue to follow up and enter the specific implementation: ffpipeline_ioc.c ---- >ffpipeline_create_from_ios

 Look at the specific implementation of ffpipeline_ioc.c ---- >func_open_video_decoder:

static IJKFF_Pipenode *func_open_video_decoder(IJKFF_Pipeline *pipeline, FFPlayer *ffp)
{
    IJKFF_Pipenode* node = NULL;
    IJKFF_Pipeline_Opaque *opaque = pipeline->opaque;
    if (ffp->videotoolbox) {
        //如果配置了硬解 即 ffp->videotoolbox 是 1 就走这里
        node = ffpipenode_create_video_decoder_from_ios_videotoolbox(ffp);
        if (!node)
            ALOGE("vtb fail!!! switch to ffmpeg decode!!!! \n");
    }
    if (node == NULL) {
        //没配置硬解 或者不支持硬解 默认使用软解
        node = ffpipenode_create_video_decoder_from_ffplay(ffp);
        ffp->stat.vdec_type = FFP_PROPV_DECODER_AVCODEC;
        opaque->is_videotoolbox_open = false;
    } else {
        //配置了硬解 并且支持硬解 就走硬解流程
        ffp->stat.vdec_type = FFP_PROPV_DECODER_VIDEOTOOLBOX;
        opaque->is_videotoolbox_open = true;
    }
    ffp_notify_msg2(ffp, FFP_MSG_VIDEO_DECODER_OPEN, opaque->is_videotoolbox_open);
    return node;
}

Since we have configured a hard solution, follow up to the ffpipenode_create_video_decoder_from_ios_videotoolbox method of the ffpipenode_ios_videotoolbox_vdec.m file to see;

IJKFF_Pipenode *ffpipenode_create_video_decoder_from_ios_videotoolbox(FFPlayer *ffp)
{
   ...
    switch (opaque->avctx->codec_id) {
    case AV_CODEC_ID_H264:
    case AV_CODEC_ID_HEVC:
        if (ffp->vtb_async){
            opaque->context = Ijk_VideoToolbox_Async_Create(ffp, opaque->avctx);
        } else {
            //这里走同步方式
            opaque->context = Ijk_VideoToolbox_Sync_Create(ffp, opaque->avctx);
        }
        break;
        ...
}

Follow up to: IJKVideoToolBox.m ----> Ijk_VideoToolbox_Sync_Create

 Follow up to: IJKVideoToolBoxSync.m ----> videotoolbox_sync_create

Ijk_VideoToolBox_Opaque* videotoolbox_sync_create(FFPlayer* ffp, AVCodecContext* avctx)
{
    ...
    //问题出在这里面
    ret = vtbformat_init(&context_vtb->fmt_desc, context_vtb->codecpar);
    ...
}

Follow up to: IJKVideoToolBoxSync.m ----> vtbformat_init

static int vtbformat_init(VTBFormatDesc *fmt_desc, AVCodecParameters *codecpar)
{
    ...
    switch (codec) {
        case AV_CODEC_ID_HEVC:
            format_id = kCMVideoCodecType_HEVC;
            if (@available(iOS 11.0, *)) {
                //iOS 11.0及以上才支持硬解 然后进一步进行判断
                isHevcSupported = VTIsHardwareDecodeSupported(kCMVideoCodecType_HEVC);
            } else {
                // Fallback on earlier versions
                isHevcSupported = false;
            }
            if (!isHevcSupported) {
                goto fail;
            }
            break;
            
        case AV_CODEC_ID_H264:
            format_id = kCMVideoCodecType_H264;
            break;
            
        default:
            goto fail;
    }
    ...
}

3. Determine the problem and solve it

Let's look at the situation of iPhone 7 and above and the iOS version is greater than 11.0:

Follow up to: IJKVideoToolBoxSync.m ----> vtbformat_init

static int vtbformat_init(VTBFormatDesc *fmt_desc, AVCodecParameters *codecpar)
{
...
    switch (codec) {
        case AV_CODEC_ID_HEVC:
            format_id = kCMVideoCodecType_HEVC;
            if (@available(iOS 11.0, *)) {
                isHevcSupported = VTIsHardwareDecodeSupported(kCMVideoCodecType_HEVC);
            } else {
                // Fallback on earlier versions
                isHevcSupported = false;
            }
            if (!isHevcSupported) {
                goto fail;
            }
            break;

    }
    
    ...
    //此处出现重大错误，
    //264的流调用ff_isom_write_avcc方法，h265（hevc）应该调用hevc.c里面的ff_isom_write_hvcc 才对        
    ff_isom_write_avcc(pb, extradata, extrasize);

    ...
    //h265（hevc）情况下不需要按264方式进行校验
    if (!validate_avcC_spc(extradata, extrasize, &fmt_desc->max_ref_frames,         &sps_level, &sps_profile)) {
            av_free(extradata);
            goto fail;
     }


}

Here's what really went wrong:

• Error 1. The ff_isom_write_avcc method is for 264 streams and cannot be used for 265 streams
• Error 2. It is not necessary to call the validate_avcC_spc method for validation under h265

Modification method:

Let me talk about error 2 first. This is relatively simple. Instead of directly performing the verification, it is performed only when it is judged to be non-hevc, as follows:

if (codec == AV_CODEC_ID_HEVC) {
    printf("---- zs log ----- vtbformat_init no check h265\n");
} else {
    if (!validate_avcC_spc(extradata, extrasize,
                    &fmt_desc->max_ref_frames, &sps_level, &sps_profile)) {
         av_free(extradata);
         printf("---- zs log ----- vtbformat_init 11 \n");
         goto fail;
    }
}

For error 1, please replace the line of code ff_isom_write_avcc(pb, extradata, extrasize); with:

//ff_isom_write_hvcc的作用是将extradata转为HEVCDecoderConfigurationRecord结构并写入。
if (codec == AV_CODEC_ID_HEVC) {
      printf("---- zs log ----- ff_isom_write_hvcc h265\n");
      ff_isom_write_hvcc(pb, extradata, extrasize, 1); 
} else {
      ff_isom_write_avcc(pb, extradata, extrasize);
}

At this time, the compilation will report an error, because the ff_isom_write_hvcc method cannot be called and is not exposed, so this method needs to be exposed first. The specific steps are as follows:

Step 1: Open ijkplayer-ios --->extra---->ffmpeg---->libavformat to get the Makefile file

  Add in the HEADERS group: hevc.h \ 

  Add hevc.o to the OBJS group \ 

 Step Two: Repeat this in the following files

• ijkplayer-ios/ios/ffmpeg-arm64/libavformat/Makefile
• ijkplayer-ios/ios/ffmpeg-armv7/libavformat/Makefile
• ijkplayer-ios/ios/ffmpeg-i386/libavformat/Makefile
• ijkplayer-ios/ios/ffmpeg-x86_64/libavformat/Makefile

Step 3: Don’t forget to add in the head of the IJKVideoToolBoxSync.m file

#include "libavformat/hevc.h"

Step 4: Recompile after clean

./compile-ffmpeg.sh clean
./compile-ffmpeg.sh all

The above is the whole modification process of ijkplayer for h265 (hevc) 4k hard solution under iOS.