Effective modeling of complex spatiotemporal dependencies in long-form videos
remains an open problem. The recently proposed Structured State-Space Sequence
(S4) model with its linear complexity offers a promising direction in this
space. However, we demonstrate that treating all image-tokens equally as done
by S4 model can adversely affect its efficiency and accuracy. To address this
limitation, we present a novel Selective S4 (i.e., S5) model that employs a
lightweight mask generator to adaptively select informative image tokens
resulting in more efficient and accurate modeling of long-term spatiotemporal
dependencies in videos. Unlike previous mask-based token reduction methods used
in transformers, our S5 model avoids the dense self-attention calculation by
making use of the guidance of the momentum-updated S4 model. This enables our
model to efficiently discard less informative tokens and adapt to various
long-form video understanding tasks more effectively. However, as is the case
for most token reduction methods, the informative image tokens could be dropped
incorrectly. To improve the robustness and the temporal horizon of our model,
we propose a novel long-short masked contrastive learning (LSMCL) approach that
enables our model to predict longer temporal context using shorter input
videos. We present extensive comparative results using three challenging
long-form video understanding datasets (LVU, COIN and Breakfast), demonstrating
that our approach consistently outperforms the previous state-of-the-art S4
model by up to 9.6% accuracy while reducing its memory footprint by 23%.

本论文提出了一种轻量级掩膜生成器的技术，采用长短掩膜对 S4 模型进行改进，从而更有效、准确地模拟视频中的长期时空依赖关系，并在三个数据集上均取得了比之前最先进的 S4 模型更高的准确性，降低了 23％的内存占用。