Posted by Xiao Xiao from Aufeisi Qubit
| Public Account QbitAI
A large model that can generate 1024×576 high-definition resolution video is open source!
Whether it is a small fish that swims among the seaweeds:
Or the mysterious Cthulhu image that is as fine as the eyeball:
All are presented with unprecedented clarity, making netizens call "the san value is crazy".
The open source video generation model exploded, not only gaining popularity on Twitter and Reddit, but even former Tesla AI director Andrej Karpathy also came to watch:
Now, Hugging Face engineers have created a trial demo, and many netizens directly show off online, such as generating a precious image of "Star Wars" Darth Vader surfing on the water:
The effect looks good too, so how exactly is it trained?
Transformation based on a large model with 1.7 billion parameters
The "prototype" of Zeroscope is a 1.7 billion-parameter Wensheng video large model that is open sourced by the ModelScope (Magic Build) community of Bodhidharma Academy.
This version of the large model consists of three sub-networks: text feature extraction, text feature to video latent space diffusion model, and video latent space to video visual space.
Among them, the diffusion model adopts Unet3D structure, and finally realizes video generation by iterative denoising process from pure Gaussian noise video.
However, for this version of the large model that is open source in the ModelScope community, the effect of generating video cannot be said to be high-definition:
To this end, ZeroScope designed two stages, first through the text-generated video, and then through the video-generated video to increase the resolution, and finally generate a 1024×576 resolution video:
The first step, Wensheng video, is based on ZeroScope_v2_576w to generate 576×320 resolution video;
The second step is to generate video from video, and generate video clips with a resolution of 1024×576 based on ZeroScope_v2_XL.
For training, ZeroScope uses 9923 1024×576 resolution video clips, each clip contains 24 frames, of which 3 frames are marked, adding up to 29769 tagged frames (tagged frames).
However, generating high-definition video requires higher hardware requirements.
To generate a video with a resolution of 576×320 and a frame rate of 30, you need at least 7.9GB of VRam (a type of video memory); if you want to generate a video with a resolution of 1024×576 and a frame rate of 30, you need at least 15.3GB of VRam .
Some netizens are happy:
Another Vinson video model that can compete with Gen-2 has appeared!
Some netizens even think that the appearance of this model has shown that people don’t need to pay for the Gen-2 made by Runway, after all, the latter effect is not so good.
In any case, the "new disruptors" in the field of Wensheng video AI have appeared.
Online demo demo is out
As soon as the model was open-sourced, a demo has already appeared on Hugging Face.
Here we try to generate "playing golf with Einstein".
The effect is not bad, although I don’t know why Einstein squatted under the name (manual dog head)
From the type of prompt words, not only can you enter a more detailed description :
For example, "A man is sleeping in his seat, inside a train running, background behind the window is moving fast" (A man is sleeping in his seat, inside a train running, background behind the window is moving fast)
You can also just enter a simple sentence , such as "Giant Pikachu versus Godzilla fight" (Giant Pikachu versus Godzilla fight)
In addition, many netizens also shared their works.
For example, here's "Einstein Laughing and Driving a Star Wars Pod":
Another netizen @Callimiya generated a magical video of "Darth Vader dancing in the classroom", and it seems that there are children dancing with him:
However, due to the large number of trial players, it sometimes has bugs. At this time, as long as you keep submitting, you can still rush into the queue.
Of course, if you feel that the controllability of this version of Demo is not good enough, you can try another version, both the seed (easy to generate similar content) and the number of inferences can be manually adjusted:
How about, have you figured out what new video to generate with it?
Simple version demo:
https://huggingface.co/spaces/fffiloni/zeroscope
Controllable version advanced demo:
https://huggingface.co/spaces/hysts/zeroscope-v2
参考链接：
[1]https://twitter.com/_akhaliq/status/1672650155743408133
[2]https://www.reddit.com/r/aivideo/comments/14hbiql/announcing_zeroscope_v2_xl_a_new_1024x576_video/
[3]https://twitter.com/fffiloni/status/1673644193967747072