Local features at neighboring spatial positions in feature maps have high
correlation since their receptive fields are often overlapped. Self-attention
usually uses the weighted sum (or other functions) with internal elements of
each local feature to obtain its weight score, which ignores interactions among
local features. To address this, we propose an effective interaction-aware
self-attention model inspired by PCA to learn attention maps. Furthermore,
since different layers in a deep network capture feature maps of different
scales, we use these feature maps to construct a spatial pyramid and then
utilize multi-scale information to obtain more accurate attention scores, which
are used to weight the local features in all spatial positions of feature maps
to calculate attention maps. Moreover, our spatial pyramid attention is
unrestricted to the number of its input feature maps so it is easily extended
to a spatio-temporal version. Finally, our model is embedded in general CNNs to
form end-to-end attention networks for action classification. Experimental
results show that our method achieves the state-of-the-art results on the
UCF101, HMDB51 and untrimmed Charades.

本研究提出一种基于主成分分析的互动感知自注意模型，通过利用多尺度信息构建空间金字塔，实现对特征图中所有空间位置的本地特征加权得到注意图，并将其嵌入卷积神经网络中以形成端到端的注意力网络，实现动作分类中的最先进结果。