FFmpeg provides audio and video filters in the libavfilter module. All video filters are registered in libavfilter/allfilters.c. We can also use the ffmpeg -filters command line to view all the currently supported filters, the front -v stands for video. This article mainly introduces video filters, including: black detection, video overlay, color balance, watermark removal, anti-shake, rectangle labeling, and Jiugongge.
For a detailed introduction to video filters, see the official document: Video Filters. For audio filters, please refer to the previous two articles: Introduction to Audio Filters (Part 1) and Introduction to Audio Filters (Part 2).
1、blackdetect
Black detection, used to detect the video interval of pure black. This filter outputs the results of its analysis to logs and metadata. If a black segment with at least the specified minimum duration is found, print a log line containing start and end timestamps and duration. The parameter options are as follows:
-
black_min_duration, d: the shortest black detection duration, in s, the default is 2.0
-
picture_black_ratio_th, pic_th: set the ratio of the black image, the default is 0.98
-
pixel_black_th, pix_th: set the threshold of black pixels, the default is 0.10
2、blend
Blending, blending all frames of two videos together, also known as video overlay. The first video is on the top layer, and the second video is on the bottom layer. By default, the longest video duration is used as the output duration.
-
2.1 Linear horizontal transition from top layer to bottom layer:
blend=all_expr='A*(X/W)+B*(1-X/W)'
-
2.2 Overlay from right to left, can be used for transition animation transition effects:
blend=all_expr='if(gte(N*SW+X,W),A,B)'
-
2.3 Overlay from top to bottom, can be used for transition animation transition effects:
blend=all_expr='if(gte(Y-N*SH,0),A,B)'
-
2.4 Divide the video along the diagonal, showing the top layer and bottom layer on both sides:
blend=all_expr='if(gt(X,Y*(W/H)),A,B)'
The video blending code is located in libavfilter/vf_blend.c, which mainly traverses the pixel matrix and calculates the sum of the top pixel multiplied by a transparency and the bottom pixel multiplied by the opposite of transparency. The key code is as follows:
static void blend_normal_8bit(const uint8_t *top, ptrdiff_t top_linesize,
const uint8_t *bottom, ptrdiff_t bottom_linesize,
uint8_t *dst, ptrdiff_t dst_linesize,
ptrdiff_t width, ptrdiff_t height,
FilterParams *param, double *values, int starty)
{
const double opacity = param->opacity;
int i, j;
for (i = 0; i < height; i++) {
for (j = 0; j < width; j++) {
dst[j] = top[j] * opacity + bottom[j] * (1. - opacity);
}
dst += dst_linesize;
top += top_linesize;
bottom += bottom_linesize;
}
}
The full command is as follows:
ffmpeg -i hello.mp4 -i world.mp4 -filter_complex blend=all_expr='A*(X/W)+B*(1-X/W)' blend.mp4
The mixing effect of the two videos is shown in the figure below:
3、colorbalance
Color balance, adjust the ratio of RGB components of the video frame. This filter allows the input frame to be adjusted in the shadows, midtones, or highlight areas for a red-cyan, magenta, or blue-yellow balance. Positive adjustment values shift the balance toward the primary color, and negative adjustment values toward the complementary color. The parameter options are as follows:
rs, gs, bs: adjust red, green, blue shadow (darkest pixel) rm, gm, bm: adjust red, green, blue tone (middle pixel) rh, gh, bh: adjust red, green, blue highlight ( brightest pixel)
4、seduction
Watermark removal, which suppresses watermarks by simple interpolation of surrounding pixels. Just set a rectangle covering the watermark. The rectangle is drawn on the outermost pixel, which is replaced by the interpolation. The value of the next pixel immediately outside the rectangle in each direction, used to compute the interpolated pixel value inside the rectangle. The parameters are as follows:
x, y: coordinates of the upper left corner of the watermark w, h: the width and height of the watermark show: whether to display the covered watermark area, the default is 0
The watermark removal code is located in libavfilter/vf_delogo.c, the core code is as follows:
static void apply_delogo(uint8_t *dst, int dst_linesize,
uint8_t *src, int src_linesize,
int w, int h, AVRational sar,
int logo_x, int logo_y, int logo_w, int logo_h,
unsigned int band, int show, int direct)
{
int x, y;
uint64_t interp, weightl, weightr, weightt, weightb, weight;
uint8_t *xdst, *xsrc;
uint8_t *topleft, *botleft, *topright;
unsigned int left_sample, right_sample;
int xclipl, xclipr, yclipt, yclipb;
int logo_x1, logo_x2, logo_y1, logo_y2;
xclipl = FFMAX(-logo_x, 0);
xclipr = FFMAX(logo_x+logo_w-w, 0);
yclipt = FFMAX(-logo_y, 0);
yclipb = FFMAX(logo_y+logo_h-h, 0);
logo_x1 = logo_x + xclipl;
logo_x2 = logo_x + logo_w - xclipr - 1;
logo_y1 = logo_y + yclipt;
logo_y2 = logo_y + logo_h - yclipb - 1;
topleft = src+logo_y1 * src_linesize+logo_x1;
topright = src+logo_y1 * src_linesize+logo_x2;
botleft = src+logo_y2 * src_linesize+logo_x1;
if (!direct)
av_image_copy_plane(dst, dst_linesize, src, src_linesize, w, h);
dst += (logo_y1 + 1) * dst_linesize;
src += (logo_y1 + 1) * src_linesize;
for (y = logo_y1+1; y < logo_y2; y++) {
left_sample = topleft[src_linesize*(y-logo_y1)] +
topleft[src_linesize*(y-logo_y1-1)] +
topleft[src_linesize*(y-logo_y1+1)];
right_sample = topright[src_linesize*(y-logo_y1)] +
topright[src_linesize*(y-logo_y1-1)] +
topright[src_linesize*(y-logo_y1+1)];
for (x = logo_x1+1,
xdst = dst+logo_x1+1,
xsrc = src+logo_x1+1; x < logo_x2; x++, xdst++, xsrc++) {
if (show && (y == logo_y1+1 || y == logo_y2-1 ||
x == logo_x1+1 || x == logo_x2-1)) {
*xdst = 0;
continue;
}
// Perform weight interpolation based on the relative distance of pixels, considering SAR
weightl = (uint64_t)(logo_x2-x) * (y-logo_y1) * (logo_y2-y) * sar.den;
weightr = (uint64_t)(x-logo_x1) * (y-logo_y1) * (logo_y2-y) * sar.den;
weightt = (uint64_t)(x-logo_x1) * (logo_x2-x) * (logo_y2-y) * sar.num;
weightb = (uint64_t)(x-logo_x1) * (logo_x2-x) * (y-logo_y1) * sar.num;
interp =
left_sample * weightl
+
right_sample * weightr
+
(topleft[x-logo_x1] +
topleft[x-logo_x1-1] +
topleft[x-logo_x1+1]) * weightt
+
(botleft[x-logo_x1] +
botleft[x-logo_x1-1] +
botleft[x-logo_x1+1]) * weightb;
weight = (weightl + weightr + weightt + weightb) * 3U;
interp = (interp + (weight >> 1)) / weight;
// Determine whether it is in the watermark area
if (y >= logo_y+band && y < logo_y+logo_h-band &&
x >= logo_x+band && x < logo_x+logo_w-band) {
*xdst = interp;
} else {
unsigned dist = 0;
if (x < logo_x+band)
dist = FFMAX(dist, logo_x-x+band);
else if (x >= logo_x+logo_w-band)
dist = FFMAX(dist, x-(logo_x+logo_w-1-band));
if (y < logo_y+band)
dist = FFMAX(dist, logo_y-y+band);
else if (y >= logo_y+logo_h-band)
dist = FFMAX(dist, y-(logo_y+logo_h-1-band));
*xdst = (*xsrc*dist + interp*(band-dist))/band;
}
}
dst += dst_linesize;
src += src_linesize;
}
}
The effect before and after removing the watermark, as shown in the following figure:
5、drawbox
Draw a rectangle, and draw a rectangle frame on the video screen, which can be used to mark the ROI area of interest. In face detection and face recognition scenarios, when a face is detected, it will be marked with a rectangular frame. The parameter options are as follows:
x, y: the xy coordinate point of the rectangle width, w, height, h: the width and height of the rectangle color, c: the color thickness of the rectangle border, t: the thickness of the rectangle border, the default is 3 Specify xy coordinates, rectangle width and height, border The color is red and the transparency is 50%, the command is as follows:
drawbox=x=10:y=20:w=200:h=60:[email protected]
6、drawgrid
Draw x-square grids, which can be used to draw four-square grids and nine-square grids, to simulate screen splicing, or screen division. The parameter options are as follows:
x, y: the xy coordinate point width of the Jiugongge, w, height, h: the width and height of each row, color, c: the thickness of the border of the Jiugongge, t: the thickness of the border of the Jiugongge, the default is 1 iw, ih: input the width and height to draw 3x3 The nine-square grid, the border thickness is 2, the color is blue and the transparency is 50%, the command is as follows:
drawgrid=w=iw/3:h=ih/3:t=2:[email protected]
7. lut, lutyuv and lutrgb
Adjust yuv or rgb, adjustment process: calculate a lookup table for binding each pixel input value to output value, and apply it to the input video. The relevant code is located in vf_lut.c, which is divided into four types for processing: packed 8bits, packed 16bits, planar 8bits, planar 16bits, the key codes are as follows:
static int filter_frame(AVFilterLink *inlink, AVFrame *in)
{
......
if (av_frame_is_writable(in)) {
direct = 1;
out = in;
} else {
// get video frame data from the buffer
out = ff_get_video_buffer(outlink, outlink->w, outlink->h);
if (!out) {
av_frame_free(&in);
return AVERROR(ENOMEM);
}
av_frame_copy_props(out, in);
}
// 分为packed 8bits、packed 16bits、planar 8bits、planar 16bits
if (s->is_rgb && s->is_16bit && !s->is_planar) {
/* packed, 16-bits */
PACKED_THREAD_DATA
ctx->internal->execute(ctx, lut_packed_16bits, &td, NULL,
FFMIN(in->height, ff_filter_get_nb_threads(ctx)));
} else if (s->is_rgb && !s->is_planar) {
/* packed 8 bits */
PACKED_THREAD_DATA
ctx->internal->execute(ctx, lut_packed_8bits, &td, NULL,
FFMIN(in->height, ff_filter_get_nb_threads(ctx)));
} else if (s->is_16bit) {
/* planar 16 bits depth */
PLANAR_THREAD_DATA
ctx->internal->execute(ctx, lut_planar_16bits, &td, NULL,
FFMIN(in->height, ff_filter_get_nb_threads(ctx)));
} else {
/* planar 8bits depth */
PLANAR_THREAD_DATA
ctx->internal->execute(ctx, lut_planar_8bits, &td, NULL,
FFMIN(in->height, ff_filter_get_nb_threads(ctx)));
}
if (!direct)
av_frame_free(&in);
return ff_filter_frame(outlink, out);
}
Convert color video to black and white video, set U and V components to 128, the reference command is as follows:
ffmpeg -i in.mp4 -vf lutyuv='u=128:v=128' gray.mp4
The effect of black and white video is shown in the figure below:
Original link: FFmpeg source code analysis: video filter introduction (Part 1)_Xu Fuji 456's Blog-CSDN Blog
★The business card at the end of the article can receive audio and video development learning materials for free, including (FFmpeg, webRTC, rtmp, hls, rtsp, ffplay, srs) and audio and video learning roadmaps, etc.
see below!