Action localization networks are often structured as a feature encoder
sub-network and a localization sub-network, where the feature encoder learns to
transform an input video to features that are useful for the localization
sub-network to generate reliable action proposals. While some of the encoded
features may be more useful for generating action proposals, prior action
localization approaches do not include any attention mechanism that enables the
localization sub-network to attend more to the more important features. In this
paper, we propose a novel attention mechanism, the Class Semantics-based
Attention (CSA), that learns from the temporal distribution of semantics of
action classes present in an input video to find the importance scores of the
encoded features, which are used to provide attention to the more useful
encoded features. We demonstrate on two popular action detection datasets that
incorporating our novel attention mechanism provides considerable performance
gains on competitive action detection models (e.g., around 6.2% improvement
over BMN action detection baseline to obtain 47.5% mAP on the THUMOS-14
dataset), and a new state-of-the-art of 36.25% mAP on the ActivityNet v1.3
dataset. Further, the CSA localization model family which includes BMN-CSA, was
part of the second-placed submission at the 2021 ActivityNet action
localization challenge. Our attention mechanism outperforms prior
self-attention modules such as the squeeze-and-excitation in action detection
task. We also observe that our attention mechanism is complementary to such
self-attention modules in that performance improvements are seen when both are
used together.

该研究提出了一种基于类语义的关注机制，用于提高行为检测模型的性能，并在两个流行的行为检测任务数据集上展示了明显的性能提升，同时对于一些自注意力模块（如收缩激活）的性能提升具有补充作用。