Our work explores the task of generating future sensor observations
conditioned on the past. We are motivated by `predictive coding' concepts from
neuroscience as well as robotic applications such as self-driving vehicles.
Predictive video modeling is challenging because the future may be multi-modal
and learning at scale remains computationally expensive for video processing.
To address both challenges, our key insight is to leverage the large-scale
pretraining of image diffusion models which can handle multi-modality. We
repurpose image models for video prediction by conditioning on new frame
timestamps. Such models can be trained with videos of both static and dynamic
scenes. To allow them to be trained with modestly-sized datasets, we introduce
invariances by factoring out illumination and texture by forcing the model to
predict (pseudo) depth, readily obtained for in-the-wild videos via
off-the-shelf monocular depth networks. In fact, we show that simply modifying
networks to predict grayscale pixels already improves the accuracy of video
prediction. Given the extra controllability with timestamp conditioning, we
propose sampling schedules that work better than the traditional autoregressive
and hierarchical sampling strategies. Motivated by probabilistic metrics from
the object forecasting literature, we create a benchmark for video prediction
on a diverse set of videos spanning indoor and outdoor scenes and a large
vocabulary of objects. Our experiments illustrate the effectiveness of learning
to condition on timestamps, and show the importance of predicting the future
with invariant modalities.

该研究旨在通过预测编码的概念以及诸如自动驾驶车辆之类的机器人应用，探索基于过去条件下生成未来传感器观测的任务。研究重点在于解决多模态未来视频建模的挑战，并利用图像扩散模型的大规模预训练来处理计算代价昂贵的视频处理问题。通过时间戳条件生成更好的采样计划，实验结果证明了学习以时间戳为条件和预测使用不变模态的重要性。