Various methods of video key frame AI

The logic of video keyframe AI is to divide the video into frames, then use SD painting to fix the style, and finally stitch them together to form a new video.

Both Mov2Mov and Multi Frame can make this kind of video. However, these operations are cumbersome. After trying to deal with a more stable method, you can pass img2im, batch_size, ControlNetperform ADetailercontrol repair and batch processing, so the operation will be very simple.

Stable diffusion method

scriptsInstall the video conversion plugin, create a convert.py file in a folder under your SD directory and copy the following code into the file.

import cv2
import gradio as gr
import os
from tqdm import tqdm
from modules import script_callbacks


def add_tab():
    with gr.Blocks(analytics_enabled=False) as ui:
        with gr.Row().style(equal_height=False):
            with gr.Column(variant='panel'):
                gr.HTML(value="<p>Extract frames from video</p>")
                input_video = gr.Textbox(label="Input video path")
                output_folder = gr.Textbox(label="Output frames folder path")
                output_fps = gr.Slider(minimum=1, maximum=120, step=1, label="Save every N frame", value=1)
                extract_frames_btn = gr.Button(label="Extract Frames", variant='primary')

            with gr.Column(variant='panel'):
                gr.HTML(value="<p>Merge frames to video</p>")
                input_folder = gr.Textbox(label="Input frames folder path")
                output_video = gr.Textbox(label="Output video path")
                output_video_fps = gr.Slider(minimum=1, maximum=60, step=1, label="Video FPS", value=30)
                merge_frames_btn = gr.Button(label="Merge Frames", variant='primary')

            extract_frames_btn.click(
                fn=extract_frames,
                inputs=[
                    input_video,
                    output_folder,
                    output_fps
                ]
            )

            merge_frames_btn.click(
                fn=merge_frames,
                inputs=[
                    input_folder,
                    output_video,
                    output_video_fps
                ]
            )

    return [(ui, "视频<->帧", "vf_converter")]


def extract_frames(video_path: str, output_path: str, custom_fps=None):
    """从视频文件中提取帧并输出为png格式

    Args:
        video_path (str): 视频文件的路径
        output_path (str): 输出帧的路径
        custom_fps (int, optional): 自定义输出帧率（默认为None，表示与视频帧率相同）

    Returns:
        None
    """
    cap = cv2.VideoCapture(video_path)
    fps = int(cap.get(cv2.CAP_PROP_FPS))
    custom_fps = int(custom_fps)
    frame_count = 0

    print(f"Extracting {
      
      video_path} to {
      
      output_path}...")
    while True:
        ret, frame = cap.read()
        if not ret:
            break

        if custom_fps:
            if frame_count % custom_fps == 0:
                output_name = os.path.join(output_path, '{:06d}.png'.format(frame_count))
                cv2.imwrite(output_name, frame)
        else:
            output_name = os.path.join(output_path, '{:06d}.png'.format(frame_count))
            cv2.imwrite(output_name, frame)
        frame_count += 1

    cap.release()
    print("Extract finished.")


def merge_frames(frames_path: str, output_path: str, fps=None):
    """将指定文件夹内的所有png图片按顺序合并为一个mp4视频文件

    Args:
        frames_path (str): 输入帧的路径（所有帧必须为png格式）
        output_path (str): 输出视频的路径（需以.mp4为文件扩展名）
        fps (int, optional): 输出视频的帧率（默认为None，表示与输入帧相同）

    Returns:
        None
    """

    # 获取所有png图片
    frames = [f for f in os.listdir(frames_path) if f.endswith('.png')]
    img = cv2.imread(os.path.join(frames_path, frames[0]))
    height, width, _ = img.shape
    fps = fps or int(cv2.CAP_PROP_FPS)

    fourcc = cv2.VideoWriter_fourcc(*'mp4v')
    video_writer = cv2.VideoWriter(output_path, fourcc, fps, (width, height))

    print(f"Merging {
      
      len(frames)} frames to video...")
    for f in tqdm(frames):
        img = cv2.imread(os.path.join(frames_path, f))
        video_writer.write(img)

    video_writer.release()
    print("Merge finished")


script_callbacks.on_ui_tabs(add_tab)

Prepare the short video that we need to perform AI processing, and be careful not to have Chinese in the path to store the video. Try to choose high-definition resolution for the video here, otherwise the final synthesized video characters will be deformed.

Open the SD to select the video and frame swap tab, fill in the video path and the path of converting frames to save pictures, the number 1 below means that every frame is intercepted, which means that if a video of 60 frames is 1 second, it will be 60 pictures.

Click to run and the picture of each frame will appear under the corresponding folder.

Select one of the pictures and drag it to img2imgthe tab, select the model and output the corresponding keywords according to your preferences. You will find that the image processed for the first time has nothing to do with the original image.

This is because ControlNet is not selected for control, click the ControlNet tab below to set Canny.

Then click Generate Image again, and you will find that the image has changed.

However, there will still be cases where the hands and feet are broken, and you need to set the hands and feet repair function in ADetailer.

After setting, click Generate again to see the effect.

Generate several times here to record the seed seed you are satisfied with, remember to save the seed and fix it.

Even if the preprocessing is completed, click next Batch批处理选项卡, and the image in ControlNet must be forked first. Then select the picture input and output path.

This operation is to batch process the images in the out folder and save them in the in folder, and then click Execute to start processing images one by one. This process is more painful for people with poor graphics cards. It takes 30 minutes to process my 576 pictures.

After all the images are generated, delete the suffixed outline image in the in folder -1.png. Go back to the video frame swap tab to set the image directory that needs to be spliced, which is the picture folder we drew with SD, and the directory where the video is saved, and select the video frame number according to the original video frame number.

After clicking Run, a video.mp4 video will be generated under the test folder, which is the result we want.

The generated video will have a flickering problem, and we will continue to update the content to solve this problem.

After Effects method

To use AE conversion, you need to download the StyleX plug-in.

Then set the parameters to preview.

The overall effect is actually incomparable with SD, it can be understood that a simple filter is used for processing. Finally, add it to the render queue output.

Ebsynth method

Ebsynth can mainly deal with the flickering problem of animation video generated by SD.

git installation addresshttps://gitcode.net/ranting8323/ebsynth_utility.git

Enter this URL in the SD extension to install it. After the installation is complete, restart SD and a tab

will appear . Enter the Ebsynth official website to download the installer according to your system. Then open it in your extension directory. The same as the SD processing method, first batch convert the video by frame, first drag the video, and then specify the directory to extract key frames. The directory of the video can also be entered manually.Ebsynth Utility

in configurationselection stage1. If you want to keep the original video size, select -1 by default. Others do not need to operate, and then click Generate. A folder is

automatically created under the project folder , and the frame-by-frame generated pictures are saved in this folder. The effect is the same as that of the previous video frame conversion. in selection . This step is to extract the extracted key frames into Ebsynth according to the interval. The key frame interval is to extract the interval according to the frame, and then click Generate. According to the configuration of the keyframe settings, extract the keyframe Key to . Go back to SD painting , and batch redraw these pictures in the same way as the Stable Diffusion method. The generated image is saved in img2img_key. Extract the mask in the selection , or not.video_frame

configurationstage2

video_key

img2img

configurationstage3.5

In configurationthe selection stage4, Extrasthe image can be enlarged or not processed.

Choose configurationto stage5place the image files processed by keyframes img2img_upscale_keyin , if the folder does not exist, create it manually. Then execute the build to generate the corresponding .ebsfiles.

Drag and drop the corresponding files in sequence after configurationselecting stage6Open . Close the mask , click , and wait for the keyframe to be generated. Processed batch images that generate keyframes.EbSynth.exe.ebsonRun All

After batch processing, click Export to AEto synthesize.