Video understanding requires the extraction of rich spatio-temporal
representations, which transformer models achieve through self-attention.
Unfortunately, self-attention poses a computational burden. In NLP, Mamba has
surfaced as an efficient alternative for transformers. However, Mamba's
successes do not trivially extend to computer vision tasks, including those in
video analysis. In this paper, we theoretically analyze the differences between
self-attention and Mamba. We identify two limitations in Mamba's token
processing: historical decay and element contradiction. We propose
VideoMambaPro (VMP) that solves the identified limitations by adding masked
backward computation and elemental residual connections to a VideoMamba
backbone. VideoMambaPro shows state-of-the-art video action recognition
performance compared to transformer models, and surpasses VideoMamba by clear
margins: 7.9% and 8.1% top-1 on Kinetics-400 and Something-Something V2,
respectively. Our VideoMambaPro-M model achieves 91.9% top-1 on Kinetics-400,
only 0.2% below InternVideo2-6B but with only 1.2% of its parameters. The
combination of high performance and efficiency makes VideoMambaPro an
interesting alternative for transformer models.

通过对自我注意力和 Mamba 之间的差异进行理论分析，我们提出了 VideoMambaPro（VMP），通过向 VideoMamba 骨干结构添加掩码反向计算和元素残差连接来解决 Mamba 的两个限制，从而实现了卓越的视频动作识别性能，超过了 transformer 模型并获得了显著的优势。