Spatiotemporal predictive learning aims to generate future frames by learning
from historical frames. In this paper, we investigate existing methods and
present a general framework of spatiotemporal predictive learning, in which the
spatial encoder and decoder capture intra-frame features and the middle
temporal module catches inter-frame correlations. While the mainstream methods
employ recurrent units to capture long-term temporal dependencies, they suffer
from low computational efficiency due to their unparallelizable architectures.
To parallelize the temporal module, we propose the Temporal Attention Unit
(TAU), which decomposes the temporal attention into intra-frame statical
attention and inter-frame dynamical attention. Moreover, while the mean squared
error loss focuses on intra-frame errors, we introduce a novel differential
divergence regularization to take inter-frame variations into account.
Extensive experiments demonstrate that the proposed method enables the derived
model to achieve competitive performance on various spatiotemporal prediction
benchmarks.

本文研究了时空预测学习方法，提出了一种通用框架，其中空间编码器和解码器捕获帧内特征，中间的时间模块捕获帧间相关性。我们提出了一种时间注意力单元（TAU）以并行化时间模块，并引入新的差异散度正则化以考虑帧间变化。广泛的实验证明，该方法使得模型在各种时空预测基准测试上具有竞争性能。