The drastic variation of motion in spatial and temporal dimensions makes the
video prediction task extremely challenging. Existing RNN models obtain higher
performance by deepening or widening the model. They obtain the multi-scale
features of the video only by stacking layers, which is inefficient and brings
unbearable training costs (such as memory, FLOPs, and training time). Different
from them, this paper proposes a spatiotemporal multi-scale model called
MS-LSTM wholly from a multi-scale perspective. On the basis of stacked layers,
MS-LSTM incorporates two additional efficient multi-scale designs to fully
capture spatiotemporal context information. Concretely, we employ LSTMs with
mirrored pyramid structures to construct spatial multi-scale representations
and LSTMs with different convolution kernels to construct temporal multi-scale
representations. Detailed comparison experiments with eight baseline models on
four video datasets show that MS-LSTM has better performance but lower training
costs.

本文介绍了一种新的视频预测模型 ——MS-LSTM，该模型采用多尺度空间和时间结构，在充分捕捉时空上下文信息方面具有高效和优秀的性能。与现有的 RNN 模型不同的是，MS-LSTM 使用 LSTMs 和不同的卷积内核设计空间和时间尺度，实现多尺度表示，有效减少了训练成本，同时取得更好的表现。