Creating a vivid video from the event or scenario in our imagination is a
truly fascinating experience. Recent advancements in text-to-video synthesis
have unveiled the potential to achieve this with prompts only. While text is
convenient in conveying the overall scene context, it may be insufficient to
control precisely. In this paper, we explore customized video generation by
utilizing text as context description and motion structure (e.g. frame-wise
depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves
joint-conditional video generation using a Latent Diffusion Model that is
pre-trained for still image synthesis and then promoted for video generation
with the introduction of temporal modules. This two-stage learning scheme not
only reduces the computing resources required, but also improves the
performance by transferring the rich concepts available in image datasets
solely into video generation. Moreover, we use a simple yet effective causal
attention mask strategy to enable longer video synthesis, which mitigates the
potential quality degradation effectively. Experimental results show the
superiority of our method over existing baselines, particularly in terms of
temporal coherence and fidelity to users' guidance. In addition, our model
enables several intriguing applications that demonstrate potential for
practical usage.

本文提出了一种使用文本作为上下文描述和动作结构（例如逐帧深度）作为具体指导的定制视频生成方法，涉及使用先前为静止图像合成预先训练的潜在扩散模型并引入时间模块进行视频生成的联合条件视频生成。实验结果表明，该方法在时间一致性和与用户指导的忠实度方面表现优异，特别是在现有基线模型方面具有更好的性能。