Compressed video action recognition has recently drawn growing attention,
since it remarkably reduces the storage and computational cost via replacing
raw videos by sparsely sampled RGB frames and compressed motion cues (e.g.,
motion vectors and residuals). However, this task severely suffers from the
coarse and noisy dynamics and the insufficient fusion of the heterogeneous RGB
and motion modalities. To address the two issues above, this paper proposes a
novel framework, namely Attentive Cross-modal Interaction Network with Motion
Enhancement (MEACI-Net). It follows the two-stream architecture, i.e. one for
the RGB modality and the other for the motion modality. Particularly, the
motion stream employs a multi-scale block embedded with a denoising module to
enhance representation learning. The interaction between the two streams is
then strengthened by introducing the Selective Motion Complement (SMC) and
Cross-Modality Augment (CMA) modules, where SMC complements the RGB modality
with spatio-temporally attentive local motion features and CMA further combines
the two modalities with selective feature augmentation. Extensive experiments
on the UCF-101, HMDB-51 and Kinetics-400 benchmarks demonstrate the
effectiveness and efficiency of MEACI-Net.

本研究提出了一种新的框架 MEACI-Net，采用 RGB 模态和运动模态，并针对压缩视频动作识别常见的问题，加入了 Motion Enhancement、Selective Motion Complement 和 Cross-Modality Augment 模块，实现了两种模态间的交互作用和特征增强，有效提高了识别准确率和计算效率。

基于关注运动增强的跨模态交互表征学习压缩视频动作识别

Representation Learning for Compressed Video Action Recognition via  Attentive Cross-modal Interaction with Motion Enhancement

For action recognition learning, 2D CNN-based methods are efficient but may
yield redundant features due to applying the same 2D convolution kernel to each
frame. Recent efforts attempt to capture motion information by establishing
inter-frame connections while still suffering the limited temporal receptive
field or high latency. Moreover, the feature enhancement is often only
performed by channel or space dimension in action recognition. To address these
issues, we first devise a Channel-wise Motion Enhancement (CME) module to
adaptively emphasize the channels related to dynamic information with a
channel-wise gate vector. The channel gates generated by CME incorporate the
information from all the other frames in the video. We further propose a
Spatial-wise Motion Enhancement (SME) module to focus on the regions with the
critical target in motion, according to the point-to-point similarity between
adjacent feature maps. The intuition is that the change of background is
typically slower than the motion area. Both CME and SME have clear physical
meaning in capturing action clues. By integrating the two modules into the
off-the-shelf 2D network, we finally obtain a Comprehensive Motion
Representation (CMR) learning method for action recognition, which achieves
competitive performance on Something-Something V1 & V2 and Kinetics-400. On the
temporal reasoning datasets Something-Something V1 and V2, our method
outperforms the current state-of-the-art by 2.3% and 1.9% when using 16 frames
as input, respectively.

本研究提出了基于 2D CNN 的动作识别方法 (CMR)，通过引入 CME 和 SME 两个 motion enhancement 模块，分别从通道和空间维度上对动态信息进行加强，进一步提高了识别准确率。在 Something-Something V1、V2 和 Kinetics-400 数据集上取得了有竞争力的性能表现。