Significant advancements in video diffusion models have brought substantial
progress to the field of text-to-video (T2V) synthesis. However, existing T2V
synthesis model struggle to accurately generate complex motion dynamics,
leading to a reduction in video realism. One possible solution is to collect
massive data and train the model on it, but this would be extremely expensive.
To alleviate this problem, in this paper, we reformulate the typical T2V
generation process as a search-based generation pipeline. Instead of scaling up
the model training, we employ existing videos as the motion prior database.
Specifically, we divide T2V generation process into two steps: (i) For a given
prompt input, we search existing text-video datasets to find videos with text
labels that closely match the prompt motions. We propose a tailored search
algorithm that emphasizes object motion features. (ii) Retrieved videos are
processed and distilled into motion priors to fine-tune a pre-trained base T2V
model, followed by generating desired videos using input prompt. By utilizing
the priors gleaned from the searched videos, we enhance the realism of the
generated videos' motion. All operations can be finished on a single NVIDIA RTX
4090 GPU. We validate our method against state-of-the-art T2V models across
diverse prompt inputs. The code will be public.

本文使用搜索为基础的生成流程，将传统的文本到视频生成过程重构，并利用先前的视频作为动作先验数据库，以此增强生成视频动作的真实感。

搜索先验使文本到视频的合成更好

Searching Priors Makes Text-to-Video Synthesis Better

Generating videos with realistic and physically plausible motion is one of
the main recent challenges in computer vision. While diffusion models are
achieving compelling results in image generation, video diffusion models are
limited by heavy training and huge models, resulting in videos that are still
biased to the training dataset. In this work we propose MotionCraft, a new
zero-shot video generator to craft physics-based and realistic videos.
MotionCraft is able to warp the noise latent space of an image diffusion model,
such as Stable Diffusion, by applying an optical flow derived from a physics
simulation. We show that warping the noise latent space results in coherent
application of the desired motion while allowing the model to generate missing
elements consistent with the scene evolution, which would otherwise result in
artefacts or missing content if the flow was applied in the pixel space. We
compare our method with the state-of-the-art Text2Video-Zero reporting
qualitative and quantitative improvements, demonstrating the effectiveness of
our approach to generate videos with finely-prescribed complex motion dynamics.
Project page: this https URL

提出了一种新的零样本视频生成器 MotionCraft，通过应用来自物理模拟的光流对图像扩散模型（如稳定扩散）的噪声潜空间进行变形，从而实现基于物理的和逼真的视频生成，以生成具有复杂运动动力学的视频。与现有方法相比，实现了定性和定量改进，证明了该方法的有效性。