1 Introduction
On November 21, Stability AI launched Stable Video Diffusion, which is Stability AI's first generative video basic model based on the image model Stable Diffusion.
Currently, Stability AI has open sourced the code of Stable Video Diffusion on GitHub, and the weights required for local operation of the model can also be found on Hugging Face.
「GitHub」
https://github.com/Stability-AI/generative-models
「Hugging Face」
https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt
Currently, the video model can be easily adapted to various downstream tasks, and it is planned to build a series of models based on this basic model, similar to building an ecosystem around stable diffusion.
In external evaluations, the two image-to-video models SVD and SVD-XT released by Stable Video Diffusion have surpassed Runway and Pika Labs in user preference studies.
2. Limitations
have to be aware of is:
1. Stability AI emphasizes that this model is not suitable for practical or commercial applications at this stage.
2. The web experience is not yet open to everyone. You can apply for a waitlist here (https://stability.ai/contact).
In addition, stable-video-diffusion-img2vid-xt has some limitations, which are also explained on Hugging Face. This model is only for research purposes.
limitation:
1. The generated video is quite short (<=4 seconds), and the model cannot achieve perfect photo-level video.
2. The current model may generate videos with no motion, or videos with very slow camera pans.
3. The model cannot be controlled through text.
4. The model cannot render clear text.
5. Faces and characters may not be generated correctly.
6. The auto-encoding part of the model is lossy.
3. How to experience
Experience address on Hugging Face: https://huggingface.co/spaces/multimodalart/stable-video-diffusion
However, after the upload image point is generated, it prompts "This application is too busy".
Later I saw a sharing from a foreign developer mkshing: https://colab.research.google.com/github/mkshing/notebooks/blob/main/stable_video_diffusion_img2vid.ipynb
After opening the link, click "Code Execution Program" - "Run All". The process is a bit slow. Interested friends can run it~
For a more detailed introduction to Stable Video Diffusion, you can view the paper: https://stability.ai/research/stable-video-diffusion-scaling-latent-video-diffusion-models-to-large-datasets
4. Reference URL
https://stability.ai/news/stable-video-diffusion-open-ai-video-model