We introduce a new attention mechanism, dubbed structural self-attention
(StructSA), that leverages rich correlation patterns naturally emerging in
key-query interactions of attention. StructSA generates attention maps by
recognizing space-time structures of key-query correlations via convolution and
uses them to dynamically aggregate local contexts of value features. This
effectively leverages rich structural patterns in images and videos such as
scene layouts, object motion, and inter-object relations. Using StructSA as a
main building block, we develop the structural vision transformer (StructViT)
and evaluate its effectiveness on both image and video classification tasks,
achieving state-of-the-art results on ImageNet-1K, Kinetics-400,
Something-Something V1 & V2, Diving-48, and FineGym.

我们引入了一种新的注意力机制，称为结构自注意力（StructSA），它利用注意力的键 - 查询交互中自然产生的丰富相关模式。使用结构自注意力作为主要构建模块，我们开发了结构视觉转换器（StructViT），并在图像和视频分类任务上评估其有效性，在 ImageNet-1K，Kinetics-400，Something-Something V1＆V2，Diving-48 和 FineGym 数据集上取得了最先进的结果。