Efficient spatiotemporal modeling is an important yet challenging problem for
video action recognition. Existing state-of-the-art methods exploit neighboring
feature differences to obtain motion clues for short-term temporal modeling
with a simple convolution. However, only one local convolution is incapable of
handling various kinds of actions because of the limited receptive field.
Besides, action-irrelated noises brought by camera movement will also harm the
quality of extracted motion features. In this paper, we propose a Temporal
Saliency Integration (TSI) block, which mainly contains a Salient Motion
Excitation (SME) module and a Cross-perception Temporal Integration (CTI)
module. Specifically, SME aims to highlight the motion-sensitive area through
spatial-level local-global motion modeling, where the saliency alignment and
pyramidal motion modeling are conducted successively between adjacent frames to
capture motion dynamics with fewer noises caused by misaligned background. CTI
is designed to perform multi-perception temporal modeling through a group of
separate 1D convolutions respectively. Meanwhile, temporal interactions across
different perceptions are integrated with the attention mechanism. Through
these two modules, long short-term temporal relationships can be encoded
efficiently by introducing limited additional parameters. Extensive experiments
are conducted on several popular benchmarks (i.e., Something-Something V1 & V2,
Kinetics-400, UCF-101, and HMDB-51), which demonstrate the effectiveness of our
proposed method.

本文提出了一种基于 Temporal Saliency Integration (TSI) 块的运动识别方法，其中主要包含了 Salient Motion Excitation (SME) 模块和 Cross-perception Temporal Integration (CTI) 模块，通过这两个模块，可以高效地编码长短时序关系。