ffmpeg-Add subtitles to video (24)

foreword

When we watch movies or videos on short video platforms such as Douyin, subtitles usually appear. With subtitles, the form of video expression is more abundant, so adding subtitles to a video is also a hard requirement. The purpose of this article is to add subtitles to a video. Before learning how to add subtitles, first understand the types of subtitles:

• External subtitle
  External subtitle is a separate external subtitle file, and the format types generally include srt, vtt, ass and so on. When playing a video, you need to put the external subtitles and the video in the same directory, and select the subtitle file in the player to see the subtitles in the video.

• Soft subtitles
  Soft subtitles are also called internal subtitles, encapsulated subtitles, internal subtitles, subtitle streams, etc. It is to embed the subtitle files of the previous external subtitles into the video as part of the stream. If a video has multiple subtitle streams, then play the video Yes, you have to select the corresponding subtitle stream

Remarks: Regardless of whether it is an external subtitle or a soft subtitle, the player must support subtitle rendering in order to display the subtitle normally.

• Hard subtitles
  Hard subtitles are subtitles embedded in the video frame. It is a part of the video frame just like a video watermark. The subtitles look the same regardless of any platform, and the player is no longer required to separately correct the letters. to render

Summary:
1. Both external subtitles and soft subtitles require the player to additionally support subtitle rendering, but hard subtitles do not. External subtitles and soft subtitles can replace and cancel subtitle files at any time, while hard subtitles cannot cancel and change subtitles in the video. 2. If it is a
subtitle stream or external subtitles, the player needs to support separate rendering of subtitle streams.
3. In addition, embedded The subtitle stream also needs container format support. For example, the MKV format supports subtitle files in various formats, but MP4 does not support subtitles very well (only Apple’s MOV text is supported)

Common Subtitle Formats

Different subtitle files have their corresponding formats (for external subtitles and soft subtitles), common subtitle formats are:

• SRT (standard external subtitle format): only contains text and time code, no style, the display effect is determined by the player, and the effect displayed by different players may vary greatly
• ASS (Advanced External Subtitle Format): Support styles, fonts, subtitle positioning, fade in and fade out, and simple special effects. If there is no shortage of fonts, the display effect of different players is basically the same
• XML+PNG sequence: used to import Premiere, FCP7, Edius, Vegas, AE, does not support FCPX
  Avid DS Cap subtitle format: AVID special format, you can modify the text after importing
• UTF (special format for VideoStudio): can be directly imported into VideoStudio for use

Recommend a subtitle production software Arctime, download address , this software can produce subtitles in various formats, the following are the formats of various subtitle files:

ass subtitle format

 

image.png

ttxt subtitle format

 

image.png

srt subtitle format

 

image.png

ffmpeg subtitle processing flow

image.png

ffmpeg command line implements adding subtitles

• Compile subtitle processing filter to ffmpeg

If ffmpeg wants to implement the function of adding subtitles, it needs to be enabled at compile time --enable-filter=subtitles --enable-libass

--enable-filter=subtitles means to enable the subtitle filter
--enable-libass is the external library that the subtitle filter needs to rely on, so you need to specify the path of the external library when compiling (like x264 compilation)

libass is an open source library for subtitle processing and rendering, address https://github.com/libass/libass.git

Complete compilation script reference: Compilation script including subtitles filter

• Add soft subtitles

 

ffmpeg -i test_1280x720_3.mp4 -i test_1280x720_3.srt -c copy output.mkv

The principle and process of adding soft subtitles is the same as adding audio to video. This process does not require recoding, so the speed is very fast.

Tips: Soft subtitles are only supported by some container formats such as (mkv), not MP4/MOV, and only some players support soft subtitles or external subtitles (such as VLC player)

VLC player plays the mkv video with soft subtitle synthesized in the above command

 

image.png

 

By default, subtitles are turned off in VLC and need to be turned on manually.

Enter the command to see that soft subtitles have been successfully added

 

ffprobe out.mkv
Input #0, matroska,webm, from '/Users/apple/devoloper/mine/ffmpeg/ffmpeg-demo/filesources/test_1280x720_3_Video_Export/out.mkv':
  Metadata:
    DESCRIPTION     : Generated by Arctime Pro 2.4
    ENCODER         : Lavf58.31.101
  Duration: 00:01:11.05, start: 0.000000, bitrate: 1435 kb/s
    Stream #0:0: Video: mpeg4 (Simple Profile), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 49.97 fps, 49.97 tbr, 1k tbn, 26635 tbc (default)
    Metadata:
      ENCODER         : Lavc58.55.100 mpeg4
      DURATION        : 00:01:11.046000000
    Stream #0:1: Audio: ac3, 44100 Hz, stereo, fltp, 192 kb/s (default)
    Metadata:
      ENCODER         : Lavc58.55.100 ac3
      DURATION        : 00:01:10.949000000
    Stream #0:2: Subtitle: ass
    Metadata:
      ENCODER         : Lavc58.55.100 ssa
      DURATION        : 00:00:18.406000000

• Subtitle format conversion
  Using the ffmpeg command can also achieve mutual conversion of subtitle formats ass/srt/vtt, etc.

 

ffmpeg -i test_1280x720_3.srt test_1280x720_3_1.vtt
ffmpeg -i test_1280x720_3.srt test_1280x720_3_1.ass

• Add hard subtitles

 

ffmpeg -i test_1280x720_3.mkv -vf subtitles=test_1280x720_3.srt out.mp4

test_1280x720_3.srt represents the path of the subtitle file to be added, and it can also be written as a subtitle file in other formats, such as test_1280x720_3.ass, test_1280x720_3.ttext and so on. In the end, ffmpeg will convert the subtitle format into an ass subtitle stream and then embed the subtitles into the video frame. This process needs to be re-encoded, so the speed is relatively slow.

Enter the command to see that hard subtitles have been successfully added

 

ffprobe out.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/apple/devoloper/mine/ffmpeg/ffmpeg-demo/filesources/test_1280x720_3_Video_Export/out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.31.101
    description     : Generated by Arctime Pro 2.4
  Duration: 00:01:11.06, start: 0.000000, bitrate: 1374 kb/s
    Stream #0:0(und): Video: mpeg4 (Simple Profile) (mp4v / 0x7634706D), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1238 kb/s, 49.97 fps, 49.97 tbr, 26635 tbn, 26635 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler

Code way to add subtitles

• 1. Add soft subtitles

 

void Subtitles::addSubtitleStream(string videopath, string spath, string dstpath)
{
    if (dstpath.rfind(".mkv") != dstpath.length() - 4) {
        LOGD("can only suport .mkv file");
        return;
    }
    
    int ret = 0;
    // 打开视频流
    if (avformat_open_input(&vfmt,videopath.c_str(), NULL, NULL) < 0) {
        LOGD("avformat_open_input failed");
        return;
    }
    if (avformat_find_stream_info(vfmt, NULL) < 0) {
        LOGD("avformat_find_stream_info");
        releaseInternal();
        return;
    }
    
    if ((avformat_alloc_output_context2(&ofmt, NULL, NULL, dstpath.c_str())) < 0) {
        LOGD("avformat_alloc_output_context2() failed");
        releaseInternal();
        return;
    }
    
    int in_video_index = -1,in_audio_index = -1;
    int ou_video_index = -1,ou_audio_index = -1,ou_subtitle_index = -1;
    for (int i=0; i<vfmt->nb_streams; i++) {
        AVStream *stream = vfmt->streams[i];
        if (stream->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
            in_video_index = i;
            AVStream *newstream = avformat_new_stream(ofmt, NULL);
            avcodec_parameters_copy(newstream->codecpar, stream->codecpar);
            newstream->codecpar->codec_tag = 0;
            ou_video_index = newstream->index;
        } else if (stream->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
            AVStream *newstream = avformat_new_stream(ofmt, NULL);
            avcodec_parameters_copy(newstream->codecpar, stream->codecpar);
            newstream->codecpar->codec_tag = 0;
            in_audio_index = i;
            ou_audio_index = newstream->index;
        }
    }
    if (!(ofmt->oformat->flags & AVFMT_NOFILE)) {
        if (avio_open(&ofmt->pb, dstpath.c_str(), AVIO_FLAG_WRITE) < 0) {
            LOGD("avio_open failed");
            releaseInternal();
            return;
        }
    }
    
    // 打开字幕流
    /** 遇到问题：调用avformat_open_input()时提示"avformat_open_input failed -1094995529(Invalid data found when processing input)"
     *  分析原因：编译ffmpeg库是没有将对应的字幕解析器添加进去比如(ff_ass_demuxer,ff_ass_muxer)
     *  解决方案：添加对应的编译参数
     */
    if ((ret = avformat_open_input(&sfmt,spath.c_str(), NULL, NULL)) < 0) {
        LOGD("avformat_open_input failed %d(%s)",ret,av_err2str(ret));
        return;
    }
    if ((ret = avformat_find_stream_info(sfmt, NULL)) < 0) {
        LOGD("avformat_find_stream_info %d(%s)",ret,av_err2str(ret));
        releaseInternal();
        return;
    }
    
    if((ret = av_find_best_stream(sfmt, AVMEDIA_TYPE_SUBTITLE, -1, -1, NULL, 0))<0){
        LOGD("not find subtitle stream 0");
        releaseInternal();
        return;
    }
    AVStream *nstream = avformat_new_stream(ofmt, NULL);
    ret = avcodec_parameters_copy(nstream->codecpar, sfmt->streams[0]->codecpar);
    nstream->codecpar->codec_tag = 0;
    /** todo:zsz AV_DISPOSITION_xxx：ffmpeg.c中该选项可以控制字幕默认是否显示，不过这里貌似不可以，原因未知。
     */
//    nstream->disposition = sfmt->streams[0]->disposition;
    ou_subtitle_index = nstream->index;
    
    if(avformat_write_header(ofmt, NULL)<0){
        LOGD("avformat_write_header failed");
        releaseInternal();
        return;
    }
    av_dump_format(ofmt, 0, dstpath.c_str(), 1);
    
    /** 遇到问题：封装后生成的mkv文件字幕无法显示，封装时提示"[matroska @ 0x10381c000] Starting new cluster due to timestamp"
     *  分析原因：通过和ffmpeg.c中源码进行比对，后发现mvk对字幕写入的顺序有要求
     *  解决方案：将字幕写入放到音视频之前
     */
    AVPacket *inpkt2 = av_packet_alloc();
    while (av_read_frame(sfmt, inpkt2) >= 0) {
        
        AVStream *srcstream = sfmt->streams[0];
        AVStream *dststream = ofmt->streams[ou_subtitle_index];
        av_packet_rescale_ts(inpkt2, srcstream->time_base, dststream->time_base);
        inpkt2->stream_index = ou_subtitle_index;
        inpkt2->pos = -1;
        LOGD("pts %d",inpkt2->pts);
        if (av_write_frame(ofmt, inpkt2) < 0) {
            LOGD("subtitle av_write_frame failed");
            releaseInternal();
            return;
        }
        av_packet_unref(inpkt2);
    }
    
    AVPacket *inpkt = av_packet_alloc();
    while (av_read_frame(vfmt, inpkt) >= 0) {
        
        if (inpkt->stream_index == in_video_index) {
            AVStream *srcstream = vfmt->streams[in_video_index];
            AVStream *dststream = ofmt->streams[ou_video_index];
            av_packet_rescale_ts(inpkt, srcstream->time_base, dststream->time_base);
            inpkt->stream_index = ou_video_index;
            LOGD("inpkt %d",inpkt->pts);
            if (av_write_frame(ofmt, inpkt) < 0) {
                LOGD("video av_write_frame failed");
                releaseInternal();
                return;
            }
        } else if (inpkt->stream_index == in_audio_index) {
            AVStream *srcstream = vfmt->streams[in_audio_index];
            AVStream *dststream = ofmt->streams[ou_audio_index];
            av_packet_rescale_ts(inpkt, srcstream->time_base, dststream->time_base);
            inpkt->stream_index = ou_audio_index;
            if (av_write_frame(ofmt, inpkt) < 0) {
                LOGD("audio av_write_frame failed");
                releaseInternal();
                return;
            }
        }
        
        av_packet_unref(inpkt);
    }
    
    LOGD("over");
    av_write_trailer(ofmt);
    releaseInternal();
    
}

Remarks:
For the encapsulation and decapsulation of mkv, the compilation parameters of ffmpeg need to be turned on --enable-muxer=matroska and --enable-demuxer=matroska.
After the subtitle ass/srt in different formats is written to the file, the subtitle will be displayed when the player opens it. The size and position also vary

• 2. Add hard subtitles

 

void Subtitles::addSubtitlesForVideo(string vpath, string spath, string dstpath,string confpath)
{
    int ret = 0;
    // 打开视频流
    if (avformat_open_input(&vfmt,vpath.c_str(), NULL, NULL) < 0) {
        LOGD("avformat_open_input failed");
        return;
    }
    if (avformat_find_stream_info(vfmt, NULL) < 0) {
        LOGD("avformat_find_stream_info");
        releaseInternal();
        return;
    }
    
    if((ret = avformat_alloc_output_context2(&ofmt, NULL, NULL, dstpath.c_str())) < 0) {
        LOGD("avformat_alloc_output_context2 failed");
        return;
    }
    
    for (int i=0; i<vfmt->nb_streams; i++) {
        AVStream *sstream = vfmt->streams[i];
        if (sstream->codecpar->codec_type == AVMEDIA_TYPE_VIDEO) {
            in_video_index = i;
            // 添加新的视频流
            AVStream *nstream = avformat_new_stream(ofmt, NULL);
            ou_video_index = nstream->index;
            
            // 由于视频需要添加字幕，所以需要重新编解码，但是编码信息和源文件中一样
            AVCodec *codec = avcodec_find_decoder(sstream->codecpar->codec_id);
            if (!codec) {
                LOGD("not surport codec!");
                releaseInternal();
                return;
            }
            de_video_ctx = avcodec_alloc_context3(codec);
            if (!de_video_ctx) {
                LOGD("avcodec_alloc_context3 failed");
                releaseInternal();
                return;
            }
            // 设置解码参数，从源文件拷贝
            avcodec_parameters_to_context(de_video_ctx, sstream->codecpar);
            // 初始化解码器上下文
            if (avcodec_open2(de_video_ctx, codec, NULL) < 0) {
                LOGD("avcodec_open2 failed");
                releaseInternal();
                return;
            }
            
            // 创建编码器
            AVCodec *encodec = avcodec_find_encoder(sstream->codecpar->codec_id);
            if (!encodec) {
                LOGD("not surport encodec!");
                releaseInternal();
                return;
            }
            en_video_ctx = avcodec_alloc_context3(encodec);
            if (!en_video_ctx) {
                LOGD("avcodec_alloc_context3 failed");
                releaseInternal();
                return;
            }
            
            // 设置编码相关参数
            /** 遇到问题：生成视频前面1秒钟是灰色的
             *  分析原因：直接从源视频流拷贝编码参数到新的编码上下文中(即通过avcodec_parameters_to_context(en_video_ctx, sstream->codecpar);)而部分重要编码参数(如帧率，时间基)并不在codecpar
             *  中，所以导致参数缺失
             *  解决方案：额外设置时间基和帧率参数
             */
            avcodec_parameters_to_context(en_video_ctx, sstream->codecpar);
            // 设置帧率
            int fps = sstream->r_frame_rate.num;
            en_video_ctx->framerate = (AVRational){fps,1};
            // 设置时间基;
            en_video_ctx->time_base = sstream->time_base;
            // I帧间隔，决定了压缩率
            en_video_ctx->gop_size = 12;
            if (ofmt->oformat->flags & AVFMT_GLOBALHEADER) {
                en_video_ctx->flags = AV_CODEC_FLAG_GLOBAL_HEADER;
            }
            // 初始化编码器上下文
            if (avcodec_open2(en_video_ctx, encodec, NULL) < 0) {
                LOGD("avcodec_open2 failed");
                releaseInternal();
                return;
            }
            
            
            // 设置视频流相关参数
            avcodec_parameters_from_context(nstream->codecpar, en_video_ctx);
            nstream->codecpar->codec_tag = 0;
            
        } else if (sstream->codecpar->codec_type == AVMEDIA_TYPE_AUDIO) {
            
            // 音频直接进行流拷贝
            in_audio_index = i;
            AVStream *nstream = avformat_new_stream(ofmt, NULL);
            avcodec_parameters_copy(nstream->codecpar, sstream->codecpar);
            ou_audio_index = nstream->index;
            nstream->codecpar->codec_tag = 0;
        }
    }
    
    if (in_video_index == -1) {
        LOGD("not has video stream");
        releaseInternal();
        return;
    }
    
    if (!(ofmt->flags & AVFMT_NOFILE)) {
        if (avio_open(&ofmt->pb, dstpath.c_str(), AVIO_FLAG_WRITE) < 0) {
            LOGD("avio_open() failed");
            releaseInternal();
            return;
        }
    }
    
    av_dump_format(ofmt, -1, dstpath.c_str(), 1);
    
    // 写入头文件
    if (avformat_write_header(ofmt, NULL) < 0) {
        LOGD("avformat_write_header failed");
        releaseInternal();
        return;
    }
    
    // 初始化滤镜
    if (!initFilterGraph(spath,confpath)) {
        LOGD("");
        releaseInternal();
        return;
    }
    
    AVPacket *inpkt = av_packet_alloc();
    while (av_read_frame(vfmt, inpkt) >= 0) {
        
        if (inpkt->stream_index == in_video_index) {
            doDecodec(inpkt);
        } else if (inpkt->stream_index == in_audio_index) {
            // 进行时间基的转换
            av_packet_rescale_ts(inpkt, vfmt->streams[in_audio_index]->time_base, ofmt->streams[ou_audio_index]->time_base);
            inpkt->stream_index = ou_audio_index;
            LOGD("audio pts %d(%s)",inpkt->pts,av_ts2timestr(inpkt->pts,&ofmt->streams[ou_audio_index]->time_base));
            av_write_frame(ofmt, inpkt);
        }
        
        av_packet_unref(inpkt);
    }
    
    LOGD("finish !");
    doDecodec(NULL);
    av_write_trailer(ofmt);
    releaseInternal();
    
}

/** 要使用subtitles和drawtext滤镜到ffmpeg中，则编译ffmpeg库时需要开启如下选项：
 *  1、字幕编解码器 --enable-encoder=ass --enable-decoder=ass --enable-encoder=srt --enable-decoder=srt --enable-encoder=webvtt --enable-decoder=webvtt；
 *  2、字幕解封装器 --enable-muxer=ass --enable-demuxer=ass --enable-muxer=srt --enable-demuxer=srt --enable-muxer=webvtt --enable-demuxer=webvtt
 *  3、滤镜选项  --enable-filter=drawtext --enable-libfreetype --enable-libass --enable-filter=subtitles
 *
 *  备注：以上字幕编解码器以及字幕解封装器可以只使用一个即可，代表只能使用一个字幕格式。具体参考编译脚本
 */
bool Subtitles::initFilterGraph(string spath,string confpath)
{
    graph = avfilter_graph_alloc();
    int ret = 0;
    AVStream *stream = vfmt->streams[in_video_index];
    // 输入滤镜
    const AVFilter *src_filter = avfilter_get_by_name("buffer");
    char desc[400];
    sprintf(desc,"video_size=%dx%d:pix_fmt=%d:time_base=%d/%d",stream->codecpar->width,stream->codecpar->height,stream->codecpar->format,stream->time_base.num,stream->time_base.den);
    ret = avfilter_graph_create_filter(&src_filter_ctx, src_filter, "buffer0", desc, NULL, graph);
    if (ret < 0) {
        LOGD("init src filter failed");
        return false;
    }

    // 输出滤镜
    const AVFilter *sink_filter = avfilter_get_by_name("buffersink");
    ret = avfilter_graph_create_filter(&sink_filter_ctx, sink_filter, "buffersink0", NULL, NULL, graph);
    if (ret < 0) {
        LOGD("buffersink init failed");
        return false;
    }
    
    /** 遇到问题：当使用libass库来合成字幕时无法生成字幕
     *  分析原因：libass使用fontconfig库来匹配字体，而程序中没有指定字体匹配用的描述文件
     *  解决方案：设置FONTCONFIG_FILE的值
     *
     *  fontconfig工作原理：fontconfig通过环境变量FONTCONFIG_FILE来找到指定的fonts.conf文件(该文件的指定了字体文件(ttf,ttc等)的目录，以及字体fallback的规则)，最终选择指定的字体文件
     *  font fallback:如果某个字符在指定的字体库中不存在，那么就需要找到能够显示此字符的备用字体库，fontconfig就是专门做此事的。
     *
     *  备注：
     *  1、mac下 系统字体库的路径为：/System/Library/Fonts
     *  2、iOS下 系统字体库的路径为：ios系统字体不允许访问
     *  3、安卓下 系统字体库的路为：/system/fonts
     *  4、Ubuntu下 系统字体库的路径为：/usr/share/fonts
     *  不同系统支持的字体库可能不一样，由于fontconfig的字体fallback机制，如果不自定义自己的字体库，可能不同系统最终因为选择的字体库不一样导致合成字幕也不一样。
     *  所以解决办法就是统一用于各个平台的字体库，然后自定义fontconfig的字体库的搜索路径
     */
    // 滤镜描述符
    setenv("FONTCONFIG_FILE",confpath.c_str(), 0);
    char filter_des[400];
    sprintf(filter_des, "subtitles=filename=%s",spath.c_str());
    AVFilterInOut *inputs = avfilter_inout_alloc();
    AVFilterInOut *ouputs = avfilter_inout_alloc();
    inputs->name = av_strdup("out");
    inputs->filter_ctx = sink_filter_ctx;
    inputs->next = NULL;
    inputs->pad_idx = 0;
    
    ouputs->name = av_strdup("in");
    ouputs->filter_ctx = src_filter_ctx;
    ouputs->next = NULL;
    ouputs->pad_idx = 0;
    
    if (avfilter_graph_parse_ptr(graph, filter_des, &inputs, &ouputs, NULL) < 0) {
        LOGD("avfilter_graph_parse_ptr failed");
        return false;
    }
    
    av_buffersink_set_frame_size(sink_filter_ctx, en_video_ctx->frame_size);
    
    // 初始化滤镜
    if (avfilter_graph_config(graph, NULL) < 0) {
        LOGD("avfilter_graph_config failed");
        return false;
    }
    
    avfilter_inout_free(&inputs);
    avfilter_inout_free(&ouputs);
    
    return true;
}

void Subtitles::doDecodec(AVPacket *pkt)
{
    if (!de_frame) {
        de_frame = av_frame_alloc();
    }
    int ret = avcodec_send_packet(de_video_ctx, pkt);
    while (true) {
        ret = avcodec_receive_frame(de_video_ctx, de_frame);
        if (ret == AVERROR_EOF) {
            // 说明已经没有数据了;清空
            //解码成功送入滤镜进行处理
            if((ret = av_buffersrc_add_frame_flags(src_filter_ctx, NULL, AV_BUFFERSRC_FLAG_KEEP_REF)) < 0) {
                LOGD("av_buffersrc_add_frame_flags failed");
                break;
            }
            break;
        } else if (ret < 0) {
            break;
        }
        
        //解码成功送入滤镜进行处理
        if((ret = av_buffersrc_add_frame_flags(src_filter_ctx, de_frame, AV_BUFFERSRC_FLAG_KEEP_REF)) < 0) {
            LOGD("av_buffersrc_add_frame_flags failed");
            break;
        }

        while (true) {
            AVFrame *enframe = av_frame_alloc();
            ret = av_buffersink_get_frame(sink_filter_ctx, enframe);
            if (ret == AVERROR_EOF) {
                // 说明结束了
                LOGD("avfilter endeof");
                // 清空编码器
                doEncodec(NULL);
                // 释放内存
                av_frame_unref(enframe);
            } else if (ret < 0) {
                // 释放内存
                av_frame_unref(enframe);
                break;
            }

            // 进行重新编码
            doEncodec(enframe);
            // 释放内存
            av_frame_unref(enframe);
        }
    }
}

void Subtitles::doEncodec(AVFrame *frame)
{
    int ret = avcodec_send_frame(en_video_ctx, frame);
    while (true) {
        AVPacket *pkt = av_packet_alloc();
        ret = avcodec_receive_packet(en_video_ctx, pkt);
        if (ret < 0) {
            av_packet_unref(pkt);
            break;
        }
        
        // 写入数据
        av_packet_rescale_ts(pkt, en_video_ctx->time_base, ofmt->streams[ou_video_index]->time_base);
        pkt->stream_index = ou_video_index;
        LOGD("video pts %d(%s)",pkt->pts,av_ts2timestr(pkt->pts,&ofmt->streams[ou_video_index]->time_base));
        av_write_frame(ofmt, pkt);
        
        av_packet_unref(pkt);
    }
}

The subtitle processing filter in ffmpeg has two subtitles and drawtext.
1. To use the subtitles filter correctly, you need to add the --enable-libass --enable-filter=subtitles configuration parameter when compiling ffmpeg, and introduce the libass library at the same time. At the same time, because the libass library references the freetype and fribidi external libraries, it is necessary to compile these two libraries at the same time. In addition, the
libass library also introduces different external libraries according to different operating systems. For example, the mac os system introduces the CoreText.framework library, Linux The fontconfig library is introduced, and the windows system introduces DirectWrite, or add --disable-require-system-font-provider
to represent the libraries that do not use these systems
. 2. To use the drawtext filter correctly, you need to add -- when compiling ffmpeg enable-filter=drawtext also needs to introduce freetype and fribidi external library
3, so the libass and drawtext filters essentially call freetype to generate a picture, and then fuse the picture and video
with the libass library subtitle processing. Libraries:
1. Text shaper related: used to define font shape related, fribidi and HarfBuzz two libraries, of which fribidi is faster, a library that has nothing to do with the font library shape, libass default, so HarfBuzz can choose not to compile 2, font
library Related: CoreText (ios/mac); fontconfig (linux/android/ios/mac); DirectWrite (windows), used to create fonts.
3. freetype: It is used to render the string into a font image according to the font and font shape specified above (RGB format, note: it can also output the RGB format as PNG, you need to compile the libpng library)

Encounter problems

1. Encountered a problem: when calling avformat_open_input(), it prompts "avformat_open_input failed -1094995529 (Invalid data found when processing input)" analysis
reason: Compiling the ffmpeg library does not add the corresponding subtitle parser, such as (ff_ass_demuxer, ff_ass_muxer)
solution : Add corresponding compilation parameters

2. Encountered a problem: the subtitles of the mkv file generated after encapsulation cannot be displayed, and the message "[matroska @ 0x10381c000] Starting new cluster due to timestamp" is prompted during encapsulation. Analysis reason: After comparing with the source code in ffmpeg.c, it is found that mvk is
not The order in which the subtitles are written is required
Solution: Write the subtitles before the audio and video

3. Encountered a problem: the first second of the generated video is gray
analysis reason: directly copy the encoding parameters from the source video stream to the new encoding context (that is, through avcodec_parameters_to_context(en_video_ctx, sstream->codecpar);) and some important encoding Parameters (such as frame rate, time base) are not in codecpar, so the parameters are missing.
Solution: additionally set time base and frame rate parameters

4. Encountered a problem: When importing fontconf into ffmpeg as a static library, it prompts "pkg-conf fontconf not found"
Analysis reason: The pc file generated by fontconf itself does not contain the expat library, which eventually leads to an error
Solution: Define fontconfig yourself library pc file

5. Encountered a problem: when importing android studio as a static library, it prompts "undefined reference to xxxx"
Analysis reason: This problem is discovered by accident, when importing an executable program as a static library (if the referenced library references other When there are mutual references between libraries or various modules), then we must pay attention to the connection sequence, so finally we must import it into android in the following order (the order of ffmpeg libraries should also be fixed) libavformat.a libavcodec.a libavfilter
. a libavutil.a libswresample.a libswscale.a libass.a libfontconfig.a libexpat.a libfreetype.a libfribidi.a libmp3lame.a libx264.a

6. Encountered a problem: "When importing fontconfig, it prompts "libtool: link: warning: library `/home/admin/usr/lib/freetype.la' was moved." "; because fontcong depends on freetype, and libass also depends on freetype. And if fontconfig adds the --with-sysroot= parameter
, the dependency_libs field of the generated fontconfig.la file is in the format of -Lxxx/freetype/lib =/user/xxxxx/freetype.la, which causes libtool parsing errors, so here fontconfig is not Need to add "--with-root" parameter

7. Encountered a problem: "Undefined symbols _libintl_dgettext" is prompted when mac is compiling.
Reason: Because the fontconfig library depends on the intl library, it is not imported during compilation.
Solution: Just import it through the compilation parameter "-lintl"

8. Encountered a problem: the real machine crashes when using the fontconfig library
Analysis reason: By looking at the source code of the fontconfig library, it is found that <Availability.h> is introduced when there is a macro definition __IPHONE_VERSION_MIN_REQUIRED in the header file fcatomic.h, so this macro is not added when compiling Definition will lead to a crash
Solution: Add macro definition __IPHONE_VERSION_MIN_REQUIRED when compiling

There are not many ffmpeg codes to complete the function of adding subtitles, and the main time is spent on solving the problems caused by the compilation and introduction of external libraries such as libass and fontconfig, so it is also recorded above

project address

https://github.com/nldzsz/ffmpeg-demo

The file Subtitles.hpp/Subtitles.cpp located in the cppsrc directory

The examples under the project can run on the iOS/android/mac platform, and the projects are located in the three directories of demo-ios/demo-android/demo-mac, and different platforms can be selected according to needs