AI video frame insertion comes with the "Video Frame Insertion" tool
视频插帧工具来啦！
DownloadThe link is at the bottom.
Preface
- Following the video cutout tool, I have considered making a video frame insertion tool. I have been fixing various problems recently (my head is getting big - _ -), but fortunately, the one that should come is finally here (^ _ ^). Nowadays, self-media is becoming more and more popular, and many people are starting to make short videos for fun. All major video platforms have also opened up high-reflection video functions. The inspiration this time came from a previous search for the current tools and methods for video frame insertion. Either I downloaded a variety of messy software, or I struggled with it for a while but the effect was still unsatisfactory. In short, it was very troublesome and a waste of time. So I still want to get a tool that can start processing with just a few clicks and can be used without too many configurations.
- Therefore, I collected some commonly used methods for frame insertion, which are basically based on the optical flow method. When I searched online, I basically used SVP4 video rendering software to increase the video to 60 frames, but this download and configuration is too troublesome. , and it’s only 60 frames. If you want to improve it, you have to spend money.
- I have read a paper "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video
Interpolation" before. So I wanted to optimize it and create a lighter model. However, the effect is still limited. The computer has been running the model for a long time, but the processing speed is still too slow. Since the nvidia graphics card used in my computer is accelerated by CUDA, most computers may only use CPU processing, so the speed is too slow. If someone only needs to process a short video, you can download it and use it.
提示：工具缺点就是处理速度较慢，还有一些功能后续再完善。
1. Video frame insertion effect
Final implementation:The number of frames is doubled.
Effect: When the picture in each frame is relatively clear, the effect is still acceptable. However, when the image is blurry and the two videos transition, the effect is still unsatisfactory.
1. Effect
Motion is generally blurry and is also the most affected part. Since the original video frames are relatively blurry, it only improves the high refresh rate blurred frames, but the look and feel is still a little more comfortable. If the original video quality is high, the effect will be will be better.
Summary: Compared with other methods, it is easier to use. However, since everyone’s computer configuration is different, there may be different bugs when transplanting to different devices. If you can use GPU acceleration, use it. If you cannot use GPU, I will not It is highly recommended to insert video frames because the processing speed is too slow. At present, a common problem with video frame insertion is processing speed, which is greatly limited by hardware. But if your computer does not support GPU acceleration and you want to try it, I also added a button to uncheck the GPU and use the CPU for processing, but it is just for entertainment.
2. Methods adopted
The author's original code is also a Unet-like structural model trained on the pytorch environment. The unsupervised model constructed based on the generated bidirectional optical flow and pseudo labels is provided here [article]. For this model, I made lightweight improvements and introduced MobileNet's separable convolution, which slightly improved the processing speed and model size, but it is still limited.
The data set used for training uses adobe240fps and is processed. The losses used include: reconstruction loss (lr), perceptual loss (lp), smoothing loss (ls) and wrap loss. For the case where GPU's CUDA acceleration is available, I use half precision for the model data to improve the processing speed, and the model structure is also slightly changed. But the arm still cannot twist the thigh. For this kind of image-generated model, it takes a lot of time.
Since it is based on the torch framework, it occupies a larger space, so this tool also occupies a larger space.
3. Usage steps
Determine whether the GPU is available
如何判断电脑是否支持GPU cuda加速:
1.首先是nvidia品牌，且cuda版本>=10.2，安装好显卡驱动。
2.其次可通过按下win+R组合键，打开cmd命令窗口。输入nvidia-smi命令，查看CUDA Version版本。
如果CUDA Version<=10.2,尝试更新显卡驱动，看看显卡是否支持更高版本的cuda。
Open Interpolation tool by CS.exe directly, select the installation location to install, and generate a desktop shortcut, which can then be opened directly.
1.Data selection
- Input Dir: Select the video to be processed;
- Output Dir: Select the save location folder;
2.Advance settings
- GPU: Select whether to use GPU. It is checked by default (if an error occurs when running, you can try to determine whether your graphics card supports cuda version 10.2 or above acceleration. If it supports, you can try to update the graphics card driver to upgrade the cuda version; if it does not support it, uncheck it and use CPU processing, but the speed is very slow);
- BatchSize: Select the batch size. Generally, if it is too large, it will burst the video memory or memory. Try to find the maximum value suitable for your computer (generally no more than 5);
- ScaleRate: The multiple of the frame rate increase. Generally, for a 30-frame video, you can choose to set it to 3 times and increase it to 90 frames (the more the number of frames is increased, the processing time will also double, so there is no need to exceed 120 frames);
3.Start Run
- ClickRun to start processing. You can view the current processing progress through the command line window (be careful not to close this window during processing).
- If you no longer want to process it, you can click theStop button to end the processing process.
The final output is output(+audio).mp4 video.
4. Solution to error message
Error 1:
Graphics card GPU cuda acceleration is not supported. According to the above method: judge whether the GPU is available and make your own judgment. If GPU is not supported, uncheck the GPU button and try running again.
Mistake 2:
The BatchSize setting is too large, which will overflow the video memory and memory. Try to adjust it smaller. If the memory is still exhausted after adjusting to 2, then the computer configuration is not good.
Mistake three:
Input Dir, Output Dir The input and output paths are not selected.
Running successfully:
Displayed respectively:Complete percentage, Elapsed running time, Estimated remaining time.
Summarize
The tool can be obtained fromNetwork Disk, extraction code: 1210
Currently there are many bugs and the processing speed is And the effect is limited, the hardware is heavy, and it is only for entertainment.
制作不易，主界面有个小彩蛋，可以打赏一下呦，感激不尽!
如果大家喜欢的话，后面可以考虑再出个增强图片、视频分辨率的工具，让你的人像与场景更加清晰！