Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -- when only future/past frames are masked; unconditional generation -- when both past and future frames are masked; and interpolation -- when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using $\le$ 4 GPUs. https://mask-cond-video-diffusion.github.io

本论文提出了一种通用框架Masked Conditional Video Diffusion（MCVD），使用基于概率条件得分的去噪扩散模型来处理所有视频合成任务，通过随机和独立的遮罩对历史或未来的帧进行条件设置，训练出一个能够执行广泛视频任务的单个模型，包括：未来/过去预测，无条件生成和插值。实验结果表明，该方法在标准视频预测和插值基准测试中产生了最先进的结果。

MCVD：基于遮蔽条件视频扩散的预测、生成和插值