Combining CNNs or ViTs, with RNNs for spatiotemporal forecasting, has yielded
unparalleled results in predicting temporal and spatial dynamics. However,
modeling extensive global information remains a formidable challenge; CNNs are
limited by their narrow receptive fields, and ViTs struggle with the intensive
computational demands of their attention mechanisms. The emergence of recent
Mamba-based architectures has been met with enthusiasm for their exceptional
long-sequence modeling capabilities, surpassing established vision models in
efficiency and accuracy, which motivates us to develop an innovative
architecture tailored for spatiotemporal forecasting. In this paper, we propose
the VMRNN cell, a new recurrent unit that integrates the strengths of Vision
Mamba blocks with LSTM. We construct a network centered on VMRNN cells to
tackle spatiotemporal prediction tasks effectively. Our extensive evaluations
show that our proposed approach secures competitive results on a variety of
tasks while maintaining a smaller model size. Our code is available at
this https URL

本文提出了一种以 VMRNN 单元为核心的网络结构，将 CNNs 和 ViTs 与 RNNs 相结合，以有效地处理时空预测任务，并通过广泛的评估表明该方法在多种任务上取得了竞争性的结果，同时保持了较小的模型尺寸。