This paper presents a novel spatiotemporal transformer network that
introduces several original components to detect actions in untrimmed videos.
First, the multi-feature selective semantic attention model calculates the
correlations between spatial and motion features to model spatiotemporal
interactions between different action semantics properly. Second, the
motion-aware network encodes the locations of action semantics in video frames
utilizing the motion-aware 2D positional encoding algorithm. Such a
motion-aware mechanism memorizes the dynamic spatiotemporal variations in
action frames that current methods cannot exploit. Third, the sequence-based
temporal attention model captures the heterogeneous temporal dependencies in
action frames. In contrast to standard temporal attention used in natural
language processing, primarily aimed at finding similarities between linguistic
words, the proposed sequence-based temporal attention is designed to determine
both the differences and similarities between video frames that jointly define
the meaning of actions. The proposed approach outperforms the state-of-the-art
solutions on four spatiotemporal action datasets: AVA 2.2, AVA 2.1, UCF101-24,
and EPIC-Kitchens.

本文介绍了一种新颖的时空变换网络，其中引入了几个原创组件以在未修剪视频中检测动作。该网络通过多特征选择性语义注意力模型计算空间和运动特征之间的关联，使用运动感知网络编码视频帧中的动作位置，并采用序列基础的时间注意力模型捕捉动作帧中的异质时间依赖关系，该方法在四个时空动作数据集上优于最先进的解决方案：AVA 2.2、AVA 2.1、UCF101-24 和 EPIC-Kitchens。