We focus on multi-modal fusion for egocentric action recognition, and propose
a novel architecture for multi-modal temporal-binding, i.e. the combination of
modalities within a range of temporal offsets. We train the architecture with
three modalities -- RGB, Flow and Audio -- and combine them with mid-level
fusion alongside sparse temporal sampling of fused representations. In contrast
with previous works, modalities are fused before temporal aggregation, with
shared modality and fusion weights over time. Our proposed architecture is
trained end-to-end, outperforming individual modalities as well as late-fusion
of modalities.
We demonstrate the importance of audio in egocentric vision, on per-class
basis, for identifying actions as well as interacting objects. Our method
achieves state of the art results on both the seen and unseen test sets of the
largest egocentric dataset: EPIC-Kitchens, on all metrics using the public
leaderboard.

提出了一种多模态融合架构，采用中层融合以及稀疏地对融合表示进行时间采样，将 RGB、Flow 和 Audio 三种模态进行融合，重点研究了多模态时间绑定，逐步改进，取得最先进的结果。