Effectively modeling long spatiotemporal sequences is challenging due to the
need to model complex spatial correlations and long-range temporal dependencies
simultaneously. ConvLSTMs attempt to address this by updating tensor-valued
states with recurrent neural networks, but their sequential computation makes
them slow to train. In contrast, Transformers can process an entire
spatiotemporal sequence, compressed into tokens, in parallel. However, the cost
of attention scales quadratically in length, limiting their scalability to
longer sequences. Here, we address the challenges of prior methods and
introduce convolutional state space models (ConvSSM) that combine the tensor
modeling ideas of ConvLSTM with the long sequence modeling approaches of state
space methods such as S4 and S5. First, we demonstrate how parallel scans can
be applied to convolutional recurrences to achieve subquadratic parallelization
and fast autoregressive generation. We then establish an equivalence between
the dynamics of ConvSSMs and SSMs, which motivates parameterization and
initialization strategies for modeling long-range dependencies. The result is
ConvS5, an efficient ConvSSM variant for long-range spatiotemporal modeling.
ConvS5 significantly outperforms Transformers and ConvLSTM on a long horizon
Moving-MNIST experiment while training 3X faster than ConvLSTM and generating
samples 400X faster than Transformers. In addition, ConvS5 matches or exceeds
the performance of state-of-the-art methods on challenging DMLab, Minecraft and
Habitat prediction benchmarks and enables new directions for modeling long
spatiotemporal sequences.

ConvSSMs combine ConvLSTM and state space methods to efficiently model long spatiotemporal sequences, outperforming Transformers and ConvLSTM in terms of training speed and sample generation while matching or exceeding state-of-the-art methods on various benchmarks.

卷积状态空间模型用于长程时空建模

Convolutional State Space Models for Long-Range Spatiotemporal Modeling

The predictive learning of spatiotemporal sequences aims to generate future
images by learning from the historical context, where the visual dynamics are
believed to have modular structures that can be learned with compositional
subsystems. This paper models these structures by presenting PredRNN, a new
recurrent network, in which a pair of memory cells are explicitly decoupled,
operate in nearly independent transition manners, and finally form unified
representations of the complex environment. Concretely, besides the original
memory cell of LSTM, this network is featured by a zigzag memory flow that
propagates in both bottom-up and top-down directions across all layers,
enabling the learned visual dynamics at different levels of RNNs to
communicate. It also leverages a memory decoupling loss to keep the memory
cells from learning redundant features. We further propose a new curriculum
learning strategy to force PredRNN to learn long-term dynamics from context
frames, which can be generalized to most sequence-to-sequence models. We
provide detailed ablation studies to verify the effectiveness of each
component. Our approach is shown to obtain highly competitive results on five
datasets for both action-free and action-conditioned predictive learning
scenarios.

本文提出了一种新的循环神经网络 PredRNN，该网络利用 zigzag 记忆流在不同层次 RNN 中进行传播，通过内存解耦丢失来防止内存单元学习重复功能，并使用课程学习策略来促使网络从上下文帧中学习长期动态。在五个数据集上进行的行为无关和行为条件预测学习场景中，该方法表现出了极高的竞争力。