Convolution has been arguably the most important feature transform for modern
neural networks, leading to the advance of deep learning. Recent emergence of
Transformer networks, which replace convolution layers with self-attention
blocks, has revealed the limitation of stationary convolution kernels and
opened the door to the era of dynamic feature transforms. The existing dynamic
transforms, including self-attention, however, are all limited for video
understanding where correspondence relations in space and time, i.e., motion
information, are crucial for effective representation. In this work, we
introduce a relational feature transform, dubbed the relational self-attention
(RSA), that leverages rich structures of spatio-temporal relations in videos by
dynamically generating relational kernels and aggregating relational contexts.
Our experiments and ablation studies show that the RSA network substantially
outperforms convolution and self-attention counterparts, achieving the state of
the art on the standard motion-centric benchmarks for video action recognition,
such as Something-Something-V1 & V2, Diving48, and FineGym.

本文提出了一种基于动态生成关系卷积核和聚合关系背景的关系特征变换 —— 关系自注意力 (RSA)，用于视频理解。通过实验和消融研究，证明 RSA 网络在视频动作识别等领域明显优于传统卷积和自注意力网络。