Spatiotemporal and motion features are two complementary and crucial
information for video action recognition. Recent state-of-the-art methods adopt
a 3D CNN stream to learn spatiotemporal features and another flow stream to
learn motion features. In this work, we aim to efficiently encode these two
features in a unified 2D framework. To this end, we first propose an STM block,
which contains a Channel-wise SpatioTemporal Module (CSTM) to present the
spatiotemporal features and a Channel-wise Motion Module (CMM) to efficiently
encode motion features. We then replace original residual blocks in the ResNet
architecture with STM blcoks to form a simple yet effective STM network by
introducing very limited extra computation cost. Extensive experiments
demonstrate that the proposed STM network outperforms the state-of-the-art
methods on both temporal-related datasets (i.e., Something-Something v1 & v2
and Jester) and scene-related datasets (i.e., Kinetics-400, UCF-101, and
HMDB-51) with the help of encoding spatiotemporal and motion features together.

本文提出了一个基于 2D 框架的 STM 网络，利用 STM 块替换 ResNet 架构中的残差块，同时编码时空特征和动作特征，实现了高效的视频动作识别。实验证明，提出的方法在数据集 Something-Something v1&v2、Jester、Kinetics-400、UCF-101 和 HMDB-51 上性能优于其他现有方法。