Our interaction with the world is an inherently multimodal experience.
However, the understanding of human-to-object interactions has historically
been addressed focusing on a single modality. In particular, a limited number
of works have considered to integrate the visual and audio modalities for this
purpose. In this work, we propose a multimodal approach for egocentric action
recognition in a kitchen environment that relies on audio and visual
information. Our model combines a sparse temporal sampling strategy with a late
fusion of audio, spatial, and temporal streams. Experimental results on the
EPIC-Kitchens dataset show that multimodal integration leads to better
performance than unimodal approaches. In particular, we achieved a 5.18%
improvement over the state of the art on verb classification.

本文提出了一种基于音频和视觉信息的厨房环境中的多模态方法，利用稀疏时间采样策略，通过音频、空间和时间流的后期融合，在 EPIC-Kitchens 数据集上实验表明多模态方法比单模态方法更好地提高了动作识别性能，特别是在动词分类上实现了 5.18% 的改进。