Tune-A-Video: One-shot Tuning of Image Diffusion Models for Text-to-Video Generation

Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation

Project: https://tuneavideo.github.io
Original Link: Tnue-A-Video: One-shot Tuning of Image Diffusion Model for Text-to-Video Generation (by Frontiers of Small Sample Vision and Intelligence)

Table of contents

01 Insufficiency of existing work?

To replicate the success of text-to-image (T2I) generation, recent work uses large-scale video datasets to train text-to-video (T2V) generators. Although their results are promising, this paradigm is computationally expensive.

02 What problem does the article solve?

We propose a new T2V generation setting - one-shot video tuning, where there is only one text-video pair. Our model builds on the state-of-the-art T2I diffusion model, which is pre-trained on a large amount of image data.

03 What is the key solution?

We introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. During inference, we employ DDIM inversion to provide structural guidance for sampling.

04 What is the main contribution?

• We introduce a new setting of One-Shot Video Tuning for T2V generation, which removes the burden of training with large-scale video datasets.
• We propose Tune-A-Video, the first framework to generate T2V using a pretrained T2I model.
• We propose efficient attention adjustment and structure inversion, which significantly improves temporal consistency.

05 What kind of related jobs are there?

• Text-to-Image diffusion models.
• Text-to-Video generative models.
• Text-driven video editing.
• Generation from a single video.

06 How is the method implemented?

Network Inflation

spatial self-attention mechanism:

Among them, $z_{v_i}$is frame $v_i$The corresponding latent code representation. $W^{\ast }$ is a learnable matrix projecting input to query, key and value, and d is the output dimension of key and query features.

We propose to use a sparse version of the causal attention mechanism, where in frame $z_{v_i}$和帧$z_{v_1}$和$z_{v_{i-1}}$Compute the attention matrix between, keeping the computational complexity low at $O(2m(N{)}^2)$.
We implement Attention(Q,k,V) as follows:

where $[\cdot ]$ indicates the connection operation, and the visual description is shown in Figure 5.

Fine-Tuning and Inference

1）Model fine-tuning

We fine-tune the entire temporal self-attention (T-Attn) layers as they are newly added. Furthermore, we propose to refine the text-video alignment (Cross-Attn) by updating the query projection in cross-attention. In practice, fine-tuning the attention block is computationally efficient compared to fully tuning [39] while preserving the original properties of the pretrained T2I diffusion model. We use the same training objective in standard ldm [37]. Figure 4 illustrates the fine-tuning process with highlighted trainable parameters.

2) Structure guidance based on DDIM inversion

The underlying noise of the source video V is obtained by DDIM inversion without text conditions. This noise serves as the starting point for DDIM sampling, given by the edited hint ${\mathcal{T}}^{\ast }$ Guidance. Output video${\mathcal{V}}^{\ast }$ is given by:

07 What are the experimental results and comparative effects?

Applications

1)Object editing.

One of the main applications of our method is to modify objects by editing text cues. This allows objects to be easily replaced, added or removed. Figure 6 shows some examples.

2）Background change.

Our method also allows the user to change the video background (i.e., where the object is located) while preserving the consistency of the object's motion. For example, we could modify the background of the skier in Figure 6 to be "on the beach" or "sunset" by adding a new location/time description and changing the country side road view in Figure 7 to an ocean view.

3)Style transfer.

Due to the open-domain knowledge of the pre-trained T2I model, our method translates videos into various styles that are difficult to learn from video data alone (12). For example, we convert real-world videos to comic-book style (Fig. 6, or Van Gogh-style (Fig. 10)) by appending a global style descriptor to the cues.

4) Personalized and controllable generation

Our method can be easily integrated with personalized T2I models (e.g., DreamBooth [39], which takes 3-5 images as input and returns a personalized T2I model), directly refining them. For example, we can use DreamBooth personalized with "Modern Disney Style" or "Mr. Potato Head" to create videos of a specific style or theme (Figure 11). Our method can also be integrated with conditional T2I models such as T2I Adapter [29] and ControlNet [52] to apply different controls on the generated videos without additional training cost. For example, we can use a sequence of human poses as controls to further edit motion (e.g., dance in Figure 1).

qualitative results

We give a visual comparison of our method with several baselines in Figure 7. In contrast, our method generates temporally coherent videos that preserve structural information in input videos and are consistent with edited words and details. Additional qualitative comparisons can be found in Figure 12.

Quantitative results

We quantify our method against baselines through automatic metrics and user studies, and report framework consistency and textual confidence in Table 1.

08 What do ablation studies tell us?

We conduct an ablation study in Tune-A-Video to evaluate the importance of the spatio-temporal attention (ST-Attn) mechanism, DDIM inversion and fine-tuning. Each design is taken individually to analyze its impact. The result is shown in Figure 8.

These results demonstrate that all of our key designs contribute to the successful results of our method.

09 How can this work be optimized?

Figure 9 shows the failure of our method when the input video contains multiple objects with occlusions. This may be due to the inherent limitations of T2I models in handling multiple objects and object interactions. A potential solution is to use additional conditional information, such as depth, to enable the model to distinguish between different objects and their interactions. Research in this area is left for future research.

10 Conclusion

In this paper, we introduce a new task generated by T2V – one-shot video tuning. This task involves training a T2V generator using only a single text-video pair and a pretrained T2I model. We present Tune-A-Video, a simple yet effective framework for text-driven video generation and editing. To generate continuous videos, we propose an efficient tuning strategy and structure inversion that can generate temporally coherent videos. Extensive experiments demonstrate that our method achieves remarkable results in a wide range of applications.

Original link: Tnue-A-Video: One-shot Tuning of Image Diffusion Model for Text-to-Video Generation (by Frontiers of Small Sample Vision and Intelligence)