Generating long, temporally consistent video remains an open challenge in
video generation. Primarily due to computational limitations, most prior
methods limit themselves to training on a small subset of frames that are then
extended to generate longer videos through a sliding window fashion. Although
these techniques may produce sharp videos, they have difficulty retaining
long-term temporal consistency due to their limited context length. In this
work, we present Temporally Consistent Video Transformer (TECO), a
vector-quantized latent dynamics video prediction model that learns compressed
representations to efficiently condition on long videos of hundreds of frames
during both training and generation. We use a MaskGit prior for dynamics
prediction which enables both sharper and faster generations compared to prior
work. Our experiments show that TECO outperforms SOTA baselines in a variety of
video prediction benchmarks ranging from simple mazes in DMLab, large 3D worlds
in Minecraft, and complex real-world videos from Kinetics-600. In addition, to
better understand the capabilities of video prediction models in modeling
temporal consistency, we introduce several challenging video prediction tasks
consisting of agents randomly traversing 3D scenes of varying difficulty. This
presents a challenging benchmark for video prediction in partially observable
environments where a model must understand what parts of the scenes to
re-create versus invent depending on its past observations or generations.
Generated videos are available at this https URL

通过引入基于向量量化的潜在动态视频预测模型和 MaskGit 先验，我们提出了一种名为 TECO 的模型，它可以在训练和生成过程中有效地处理数百帧的长视频以实现长期时间一致性，不仅在简单的迷宫和大型 3D 世界中，而且在真实世界的复杂视频中也优于 SOTA 基准。