Convolutional operations have two limitations: (1) do not explicitly model
where to focus as the same filter is applied to all the positions, and (2) are
unsuitable for modeling long-range dependencies as they only operate on a small
neighborhood. While both limitations can be alleviated by attention operations,
many design choices remain to be determined to use attention, especially when
applying attention to videos. Towards a principled way of applying attention to
videos, we address the task of spatiotemporal attention cell search. We propose
a novel search space for spatiotemporal attention cells, which allows the
search algorithm to flexibly explore various design choices in the cell. The
discovered attention cells can be seamlessly inserted into existing backbone
networks, e.g., I3D or S3D, and improve video classification accuracy by more
than 2% on both Kinetics-600 and MiT datasets. The discovered attention cells
outperform non-local blocks on both datasets, and demonstrate strong
generalization across different modalities, backbones, and datasets. Inserting
our attention cells into I3D-R50 yields state-of-the-art performance on both
datasets.

提出一种新的用于搜索空间时间关注单元的方法，可以用于改善现有的骨干网络（如 I3D 或 S3D）中的视频分类精度，并在 Kinetics-600 和 MiT 数据集上超过 2％。