This article is based on ffmpeg4.4 source code.
a.mp4 download link: Baidu Netdisk , extraction code: nl0s. logo.jpg Address: Click to view
The command is as follows:
ffmpeg.exe -i a.mp4 -i logo.jpg -filter_complex "[1:v]scale=176:144[logo];[0:v][logo]overlay=x=0:y=0" output.mp4 -y
The function achieved by the above command is to place the logo of "Overtones" in the upper left corner of the video.
The ffmpeg command line has two filter usages:
1, -vf, ordinary filter, explained in " ffmpeg command analysis -vf ".
What is a simple filter? Only one input stream is a simple filter
2, -filter_complex, -lavfi the parameters of these two commands are the same, this is a complex filter, and lavfi is estimated to be the abbreviation of libavfilter.
What are complex filters? Complex filters have multiple input streams. The command in this article has 2 input streams, which belong to complex filters.
Complex filters are the focus of this article.
First filter_complex is defined in ffmpeg_opt.c as follows:
{ "filter_complex", HAS_ARG | OPT_EXPERT, { .func_arg = opt_filter_complex },
"create a complex filtergraph", "graph_description" }It can be seen from the definition that filter_complex will call the opt_filter_complex function.
The opt_filter_complex function is defined as follows:
static int opt_filter_complex(void *optctx, const char *opt, const char *arg)
{
GROW_ARRAY(filtergraphs, nb_filtergraphs);
if (!(filtergraphs[nb_filtergraphs - 1] = av_mallocz(sizeof(*filtergraphs[0]))))
return AVERROR(ENOMEM);
filtergraphs[nb_filtergraphs - 1]->index = nb_filtergraphs - 1;
filtergraphs[nb_filtergraphs - 1]->graph_desc = av_strdup(arg);
if (!filtergraphs[nb_filtergraphs - 1]->graph_desc)
return AVERROR(ENOMEM);
input_stream_potentially_available = 1;
return 0;
}As can be seen from the above code, what opt_filter_complex does is very simple, that is, malloc one struct FilterGraph, and then put it in the global variable filtergraphs.
The parameter string after -filter_complex "[1:v]scale=176:144[logo];[0:v][logo]overlay=x=0:y=0" is saved in graph_desc.
ffmpeg will judge whether the FilterGraph is a complex FilterGraph through the graph_desc parameter, and filtergraph_is_simple() realize it through a function.
In the previous article " ffmpeg source code analysis - open_output_file ", we know init_simple_filtergraphthat the function is executed in open_output_file.
init_simple_filtergraph is to initialize a simple filter . init_complex_filters is to initialize complex filters , and the calling process between them is as follows:
The flow chart is as follows:
As can be seen from the above flowchart, init_complex_filters is executed before init_simple_filtergraph. If init_complex_filters is executed, init_simple_filtergraph will not be executed, and only one will be executed. For example, if the video filter uses init_complex_filters, init_simple_filtergraph will not be executed, but the audio in this command The filter will be initialized with init_simple_filtergraph.
The logic of init_complex_filters is carefully analyzed below.
The init_complex_filters function code is as follows:
static int init_complex_filters(void)
{
int i, ret = 0;
for (i = 0; i < nb_filtergraphs; i++) {
ret = init_complex_filtergraph(filtergraphs[i]);
if (ret < 0)
return ret;
}
return 0;
}This function is relatively simple, which is to execute init_complex_filtergraph in a loop. The command in this article has only one complex filter, so it can only loop once. The loop here is actually for processing that kind of complex filter.
Then analyze the logic of the init_complex_filtergraph function, the key points are as follows:
As can be seen from the above figure, the init_complex_filtergraph() function calls avfilter_graph_parse2() to parse "[1:v]scale=176:144[logo];[0:v][logo]overlay=x=0:y=0" .
Note here that the third parameter, the inputs variable, is an struct AVFilterInOut array. From the debugger, we can see that the inputs array has two values, 1:v which 0:vmatch the command line parameters.
It means that this filter-graph has two input streams. This is different from the previous article analysis. The previous filter article analysis only talked about the situation of one input stream.
Then analyze what is done in the init_input_filter() function, the flow chart is as follows:
The code of init_input_filter is a bit long, and only some key codes are posted for explanation.
From the flow chart and the code, it can be seen that the first half of the init_input_filter function is to find the ist, which is one struct InputStreamand placed in the global variable input_streams.
The key codes are as follows:
//找出 文件 ctx
s = input_files[file_idx]->ctx;
for (i = 0; i < s->nb_streams; i++) {
enum AVMediaType stream_type = s->streams[i]->codecpar->codec_type;
if (stream_type != type &&
!(stream_type == AVMEDIA_TYPE_SUBTITLE &&
type == AVMEDIA_TYPE_VIDEO /* sub2video hack */))
continue;
if (check_stream_specifier(s, s->streams[i], *p == ':' ? p + 1 : p) == 1) {
//找出 指定流，重点
st = s->streams[i];
break;
}
}
//找出 InputStream
ist = input_streams[input_files[file_idx]->ist_index + st->index];
The init_input_filter function will extract in->name at the beginning. In this command, in->name is 1:v , its logic will extract 1 and assign it to file_idx, because the subscript starts with 0, so 1 here is specified Then extract ":v" from the second file, use check_stream_specifier to get the specified stream, v means video stream, assign the first video stream of the file to st, if the file has multiple video streams, Only the first video stream is taken, and the others are ignored.
So [1:v] in the command line parameter is to specify the first video stream of the second input file.
Finally by st = input_streams[input_files[file_idx]->ist_index + st->index] getting the InputStream.
Continue to analyze, followed by adding and initializing fg->inputs, the code is as follows:
GROW_ARRAY(fg->inputs, fg->nb_inputs);
if (!(fg->inputs[fg->nb_inputs - 1] = av_mallocz(sizeof(*fg->inputs[0]))))
exit_program(1);
//重点代码
fg->inputs[fg->nb_inputs - 1]->ist = ist;
fg->inputs[fg->nb_inputs - 1]->graph = fg;
fg->inputs[fg->nb_inputs - 1]->format = -1;
fg->inputs[fg->nb_inputs - 1]->type = ist->st->codecpar->codec_type;
fg->inputs[fg->nb_inputs - 1]->name = describe_filter_link(fg, in, 1);
fg->inputs[fg->nb_inputs - 1]->frame_queue = av_fifo_alloc(8 * sizeof(AVFrame*));
if (!fg->inputs[fg->nb_inputs - 1]->frame_queue)
exit_program(1);The above code has three important points:
1. This filter has two input streams, but there is no sequence distinction, and there is no field to store which is 0 and which is 1.
2. fg->inputs[fg->nb_inputs - 1]->ist = ist; This sentence is the key code. InputFilter and InputStream are associated here.
3. The frame_queue in strcut InputFilter is a temporary storage area for AVFrame. Why do you need temporary storage? It is because all InputFilters in FilterGraph have been initialized before writing AVframe into a certain filter. This is how ffmpeg judges whether InputFilter is initialized. InputFilter If ::format is not equal to -1, the initialization is complete. The specific implementation is in the ifilter_has_all_input_formats() function. If the initialization of InputFilter A is completed, but the initialization of InputFilter B is not completed, it will not write data to InputFilter::filter of A, but first write to InputFilter::frame_queue of A, and then take it out from InputFilter::frame_queue and write to InputFilter::filter. Part of the code is as follows:
//ffmpeg.c 2213行
if (!ifilter_has_all_input_formats(fg)) {
AVFrame *tmp = av_frame_clone(frame);
if (!tmp)
return AVERROR(ENOMEM);
av_frame_unref(frame);
if (!av_fifo_space(ifilter->frame_queue)) {
ret = av_fifo_realloc2(ifilter->frame_queue, 2 * av_fifo_size(ifilter->frame_queue));
if (ret < 0) {
av_frame_free(&tmp);
return ret;
}
}
av_fifo_generic_write(ifilter->frame_queue, &tmp, sizeof(tmp), NULL);
return 0;
}PS: I don’t have the command scenario for the above code. What kind of command will run the above temporary storage logic, bury a hole, and fill in it later. If you have friends who know it, you can pay attention to it in the article evaluation.
The init_input_filter function contains an important point, the last two lines of code are as follows:
GROW_ARRAY(ist->filters, ist->nb_filters);
ist->filters[ist->nb_filters - 1] = fg->inputs[fg->nb_inputs - 1];This is to synchronize the filters in the InputStream. I don't know why I do this. I want to remember that it should be used somewhere.
So far, the init_input_filter function analysis is completed, and then the logic behind the upper layer function init_complex_filtergraph is analyzed, the code is as follows:
for (cur = outputs; cur;) {
GROW_ARRAY(fg->outputs, fg->nb_outputs);
fg->outputs[fg->nb_outputs - 1] = av_mallocz(sizeof(*fg->outputs[0]));
if (!fg->outputs[fg->nb_outputs - 1])
exit_program(1);
fg->outputs[fg->nb_outputs - 1]->graph = fg;
fg->outputs[fg->nb_outputs - 1]->out_tmp = cur;
fg->outputs[fg->nb_outputs - 1]->type = avfilter_pad_get_type(cur->filter_ctx->output_pads,
cur->pad_idx);
fg->outputs[fg->nb_outputs - 1]->name = describe_filter_link(fg, cur, 0);
cur = cur->next;
fg->outputs[fg->nb_outputs - 1]->out_tmp->next = NULL;
}The above code is actually the operation and processing of fg->outputs. The command outputs array in this article has only one value, so it will only loop once. This code is relatively easy to understand and does not require much analysis.
At this point, the analysis of the init_complex_filtergraph function is complete.
As can be seen from the previous flowchart, init_complex_filters is called before init_simple_filtergraph. Here we will focus on explaining the difference between init_complex_filters and init_simple_filtergraph.
init_complex_filters is used to initialize complex filters, what is a complex filter? A filter with multiple input streams is a complex filter.
init_simple_filtergraph is used to initialize a simple filter, what is a simple filter, only one input stream is a simple filter.
For video, if init_complex_filters is used to initialize the filter, the code will not execute init_simple_filtergraph, and only one of the two will be executed.
For example, in this command, the video filter is realized by init_complex_filters, and the audio filter is realized by calling init_simple_filtergraph, the code is as follows:
/* create streams for all unlabeled output pads */
for (i = 0; i < nb_filtergraphs; i++) {
FilterGraph *fg = filtergraphs[i];
for (j = 0; j < fg->nb_outputs; j++) {
OutputFilter *ofilter = fg->outputs[j];
if (!ofilter->out_tmp || ofilter->out_tmp->name)
continue;
switch (ofilter->type) {
case AVMEDIA_TYPE_VIDEO: o->video_disable = 1; break;
case AVMEDIA_TYPE_AUDIO: o->audio_disable = 1; break;
case AVMEDIA_TYPE_SUBTITLE: o->subtitle_disable = 1; break;
}
init_output_filter(ofilter, o, oc);
}
}The above code will set o->video_disable to 1, resulting in init_simple_filtergraph not executing.
Here is an important point, because the video output stream corresponds to multiple input streams, as shown in the figure below, so ost->source_index it will be set to -1 in the init_output_filter function, because it is not a single input stream.
When parsing complex filter parameters at the beginning, a FilterGraph has been inserted into the global variable filtergraphs array, and then when audio is processed in the open_output_file function, init_simple_filtergraph is executed, and another FilterGraph is inserted. So now the data structure looks like this:
PS: init_simple_filtergraph is explained in " ffmpeg source code analysis-open_output_file ". In this command, the audio filter is an empty FilterGraph.
Pay attention to the structure diagram above, the complex filter has nb_input equal to 2, which means there are two input streams.
After executing init_complex_filters and init_simple_filtergraph to initialize simple and complex filtergraphs, the configure_filtergraph() function will be executed later. Let’s analyze the logic of configure_filtergraph in this command. The code is shown in the figure below: the key points have been circled.
if (simple) {
//简单 filter 处理 省略
} else {
fg->graph->nb_threads = filter_complex_nbthreads;
}As can be seen from the above code, the configure_filtergraph entry is to distinguish between simple and complex filters at the beginning, and the logic of complex filters is relatively simple here.
Then avfilter_graph_parse2 was executed again, why did I say "again" , please pay attention, at the beginning, avfilter_graph_parse2 has been executed once in the init_complex_filtergraph function. For complex filters only, this concept is the most important.
The reason why complex filters execute avfilter_graph_parse2 twice is not because of wrong code, but necessary.
The first time avfilter_graph_parse2 is to get out FilterGraph::InputFilter and FilterGraph::OutputFilter, and get these two things right.
The second avfilter_graph_parse2 is for the later configure_input_filter (the third red circle).
The second avfilter_graph_parse2 is common to simple filters and complex filters, so it is executed twice, which is actually for general use. If we adjust the API function, even for complex filters, we can only adjust avfilter_graph_parse2 once.
There is another important point, the code is as follows:
for (cur = inputs, i = 0; cur; cur = cur->next, i++)
if ((ret = configure_input_filter(fg, fg->inputs[i], cur)) < 0) {
avfilter_inout_free(&inputs);
avfilter_inout_free(&outputs);
goto fail;
}The above for loop will loop twice. It does not use subscript positioning because the order of the inputs array returned by avfilter_graph_parse2 is the same twice.
The key code of configure_input_filter function is as follows:
if ((ret = avfilter_graph_create_filter(&ifilter->filter, buffer_filt, name,
args.str, NULL, fg->graph)) < 0)
goto fail;Initialize ifilter->filter as buffer filter (entry filter), then insert some default filters, such as trim filter, and finally associate it with the incoming cur variable.
The configure_filtergraph function analysis is complete.
It can be seen that the implementation of complex filters in ffmpeg.c is indeed complicated. But from the perspective of API functions, the api functions used by complex filters and simple filters are the same, both use avfilter_graph_parse2.
The complexity in the complex filter is to make the command line parameters easier to use.
The complex filter calls avfilter_graph_parse2 for the first time to handle the relationship between filter and stream. It is stated in the program that a certain InputStream needs to be sent to a certain InputFilter.
The second call to avfilter_graph_parse2 really starts to associate InputFilter with OutputFilter, because some other filters may need to be inserted in the middle, such as trim filter, rotate filter and so on.
If you adjust the api function yourself, in our code, which InputStream needs to be sent to which InputFilter has been determined, and there is no need to pass the command
To change the parameters, you can adjust avfilter_graph_parse2 once, and you are done, then read AVFrame from different streams, and then send AVFrame to different buffer filters.
Read the author's blog , Beyond the Strings , to learn more about audio and video techniques.
Due to the limited level of the author and the need to participate in the development work while writing, there will inevitably be some mistakes or inaccuracies in the article. Readers are kindly requested to criticize and correct. If readers have any valuable comments, you can add me on WeChat Loken1.
Analysis of the original ffmpeg command - filter_complex - Programmer Sought
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!