Recently there has been a lot of progress in text-to-video generation, with
state-of-the-art models being capable of generating high quality, realistic
videos. However, these models lack the capability for users to interactively
control and generate videos, which can potentially unlock new areas of
application. As a first step towards this goal, we tackle the problem of
endowing diffusion-based video generation models with interactive
spatio-temporal control over their output. To this end, we take inspiration
from the recent advances in segmentation literature to propose a novel
spatio-temporal masked attention module - Peekaboo. This module is a
training-free, no-inference-overhead addition to off-the-shelf video generation
models which enables spatio-temporal control. We also propose an evaluation
benchmark for the interactive video generation task. Through extensive
qualitative and quantitative evaluation, we establish that Peekaboo enables
control video generation and even obtains a gain of upto 3.8x in mIoU over
baseline models.

最近，在文本到视频生成方面取得了很大的进展，目前的模型能够生成高质量、逼真的视频。然而，这些模型缺乏用户交互控制和生成视频的能力，这可能开启新的应用领域。本论文提出了一种通过扩展性 - 自适应控制来加强扩散型视频生成模型的输出的方法。该模块是一种训练免费、没有推理开销的自定义部件，可以与现有的视频生成模型相结合，实现时空控制。我们还提出了一个用于交互式视频生成任务的评估基准。通过广泛的定性和定量评估，我们证明了 Peekaboo 可以实现视频生成控制，并且在 mIoU 方面超过基准模型的增益高达 3.8 倍。