Diffusion-based text-to-video generation has witnessed impressive progress in
the past year yet still falls behind text-to-image generation. One of the key
reasons is the limited scale of publicly available data (e.g., 10M video-text
pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost
of video captioning. Instead, it could be far easier to collect unlabeled clips
from video platforms like YouTube. Motivated by this, we come up with a novel
text-to-video generation framework, termed TF-T2V, which can directly learn
with text-free videos. The rationale behind is to separate the process of text
decoding from that of temporal modeling. To this end, we employ a content
branch and a motion branch, which are jointly optimized with weights shared.
Following such a pipeline, we study the effect of doubling the scale of
training set (i.e., video-only WebVid10M) with some randomly collected
text-free videos and are encouraged to observe the performance improvement (FID
from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of
our approach. We also find that our model could enjoy sustainable performance
gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some
text labels for training. Finally, we validate the effectiveness and
generalizability of our ideology on both native text-to-video generation and
compositional video synthesis paradigms. Code and models will be publicly
available at this https URL

通过使用无文本视频进行训练，研究表明，扩大训练集规模并重新引入部分文本标签，可以使基于扩散的文本到视频生成的性能得到提升和改进。

文本无关视频生成的扩展方法

A Recipe for Scaling up Text-to-Video Generation with Text-free Videos

Recently, video generation has attracted massive attention and yielded
noticeable outcomes. Concerning the characteristics of video, multi-text
conditioning incorporating sequential events is necessary for next-step video
generation. In this work, we propose a novel multi-text video generation~(MTVG)
by directly utilizing a pre-trained diffusion-based text-to-video~(T2V)
generation model without additional fine-tuning. To generate consecutive video
segments, visual consistency generated by distinct prompts is necessary with
diverse variations, such as motion and content-related transitions. Our
proposed MTVG includes Dynamic Noise and Last Frame Aware Inversion which
reinitialize the noise latent to preserve visual coherence between videos of
different prompts and prevent repetitive motion or contents. Furthermore, we
present Structure Guiding Sampling to maintain the global appearance across the
frames in a single video clip, where we leverage iterative latent updates
across the preceding frame. Additionally, our Prompt Generator allows for
arbitrary format of text conditions consisting of diverse events. As a result,
our extensive experiments, including diverse transitions of descriptions,
demonstrate that our proposed methods show superior generated outputs in terms
of semantically coherent and temporally seamless video.Video examples are
available in our project page: this https URL

最近，视频生成引起了广泛关注并取得了显着成果。针对视频的特点，多文本条件在下一步视频生成中需要结合顺序事件。本研究提出了一种新的多文本视频生成模型，通过直接利用预训练的基于扩散的文本到视频转换模型进行生成，而无需额外的微调。为了生成连续的视频片段，不同提示生成的视觉一致性是必要的，具有多样的变化，如运动和内容相关的过渡。我们的方法包括动态噪声和最后一帧感知反演，用于在不同提示的视频之间重新初始化噪声潜变量，以保持视觉一致性并防止重复运动或内容。此外，我们提出了结构导向采样，以在单个视频剪辑的帧之间保持全局外观，其中我们通过对前一帧进行迭代潜变量更新。此外，我们的提示生成器允许由各种事件组成的文本条件的任意格式。我们的广泛实验证明了我们的方法在语义上的一致性和时间上的连续性方面具有出色的生成输出。项目页面提供了视频示例：[此链接](https://this_URL)