Recognizing human actions in videos requires spatial and temporal
understanding. Most existing action recognition models lack a balanced
spatio-temporal understanding of videos. In this work, we propose a novel
two-stream architecture, called Cross-Attention in Space and Time (CAST), that
achieves a balanced spatio-temporal understanding of videos using only RGB
input. Our proposed bottleneck cross-attention mechanism enables the spatial
and temporal expert models to exchange information and make synergistic
predictions, leading to improved performance. We validate the proposed method
with extensive experiments on public benchmarks with different characteristics:
EPIC-KITCHENS-100, Something-Something-V2, and Kinetics-400. Our method
consistently shows favorable performance across these datasets, while the
performance of existing methods fluctuates depending on the dataset
characteristics.

我们提出了一种新的双流架构，称为 Cross-Attention in Space and Time (CAST)，它使用 RGB 输入实现了对视频的平衡的时空理解。我们的提出的瓶颈交叉注意机制使得空间和时间专家模型能够交换信息并进行协同预测，从而提高性能。我们通过对 EPIC-KITCHENS-100、Something-Something-V2 和 Kinetics-400 等公共基准数据集进行了大量实验证明了所提方法的优越性能。与现有方法相比，在不同数据集特征下，我们的方法始终表现出较好的性能。