Humans naturally perceive surrounding scenes by unifying sound and sight in a
first-person view. Likewise, machines are advanced to approach human
intelligence by learning with multisensory inputs from an egocentric
perspective. In this paper, we explore the challenging egocentric audio-visual
object localization task and observe that 1) egomotion commonly exists in
first-person recordings, even within a short duration; 2) The out-of-view sound
components can be created while wearers shift their attention. To address the
first problem, we propose a geometry-aware temporal aggregation module to
handle the egomotion explicitly. The effect of egomotion is mitigated by
estimating the temporal geometry transformation and exploiting it to update
visual representations. Moreover, we propose a cascaded feature enhancement
module to tackle the second issue. It improves cross-modal localization
robustness by disentangling visually-indicated audio representation. During
training, we take advantage of the naturally available audio-visual temporal
synchronization as the ``free'' self-supervision to avoid costly labeling. We
also annotate and create the Epic Sounding Object dataset for evaluation
purposes. Extensive experiments show that our method achieves state-of-the-art
localization performance in egocentric videos and can be generalized to diverse
audio-visual scenes.

本文提出了一个几何感知的时间聚合模块和级联特征增强模块来解决显式处理自我运动和消除视角移动对音视觉定位的影响，并通过自我监督学习开发 Epic Sounding Object 数据集评估模型，证明了我们的方法在以人为中心的视频中取得了最先进的对象定位性能，并可以推广到不同的音视觉场景。