In this paper, we propose a new Transformer block for video future frames
prediction based on an efficient local spatial-temporal separation attention
mechanism. Based on this new Transformer block, a fully autoregressive video
future frames prediction Transformer is proposed. In addition, a
non-autoregressive video prediction Transformer is also proposed to increase
the inference speed and reduce the accumulated inference errors of its
autoregressive counterpart. In order to avoid the prediction of very similar
future frames, a contrastive feature loss is applied to maximize the mutual
information between predicted and ground-truth future frame features. This work
is the first that makes a formal comparison of the two types of attention-based
video future frames prediction models over different scenarios. The proposed
models reach a performance competitive with more complex state-of-the-art
models. The source code is available at \emph{this https URL}.

本文提出了一个基于局部时空分离的有效空间 - 时间注意机制的 Transformer 块，用于视频未来帧预测，并构建了一个全自回归视频未来帧预测 Transformer 框架，另外还提出了一个非自回归视频预测 Transformer 框架，并引入对比特征损失来监督模型预测过程。本文是第一个在不同场景下对这两种基于注意力的视频未来帧预测模型进行正式比较的工作，所提出的模型在性能上与更复杂的现有模型竞争力相当。