Action recognition has become one of the popular research topics in computer vision. There are various methods based on Convolutional Networks and self-attention mechanisms as Transformers to solve both spatial and temporal dimensions problems of action recognition tasks that achieve competitive performances. However, these methods lack a guarantee of the correctness of the action subject that the models give attention to, i.e., how to ensure an action recognition model focuses on the proper action subject to make a reasonable action prediction. In this paper, we propose a multi-view attention consistency method that computes the similarity between two attentions from two different views of the action videos using Directed Gromov-Wasserstein Discrepancy. Furthermore, our approach applies the idea of Neural Radiance Field to implicitly render the features from novel views when training on single-view datasets. Therefore, the contributions in this work are three-fold. Firstly, we introduce the multi-view attention consistency to solve the problem of reasonable prediction in action recognition. Secondly, we define a new metric for multi-view consistent attention using Directed Gromov-Wasserstein Discrepancy. Thirdly, we built an action recognition model based on Video Transformers and Neural Radiance Fields. Compared to the recent action recognition methods, the proposed approach achieves state-of-the-art results on three large-scale datasets, i.e., Jester, Something-Something V2, and Kinetics-400.

我们提出了一种多视角注意力一致性方法，使用有向Gromov-Wasserstein差异计算来自行动视频的两个不同视角的注意力之间的相似度。此外，在单视角数据集上训练时，我们的方法应用了神经辐射场的思想来隐式渲染来自新视角的特征。与最近的行动识别方法相比，我们提出的方法在三个大规模数据集（Jester，Something-Something V2和Kinetics-400）上取得了最先进的结果。

基于有向 Gromov-Wasserstein 差异的多视角动作识别