Humans are able to localize objects in the environment using both visual and
auditory cues, integrating information from multiple modalities into a common
reference frame. We introduce a system that can leverage unlabeled audio-visual
data to learn to localize objects (moving vehicles) in a visual reference
frame, purely using stereo sound at inference time. Since it is labor-intensive
to manually annotate the correspondences between audio and object bounding
boxes, we achieve this goal by using the co-occurrence of visual and audio
streams in unlabeled videos as a form of self-supervision, without resorting to
the collection of ground-truth annotations. In particular, we propose a
framework that consists of a vision "teacher" network and a stereo-sound
"student" network. During training, knowledge embodied in a well-established
visual vehicle detection model is transferred to the audio domain using
unlabeled videos as a bridge. At test time, the stereo-sound student network
can work independently to perform object localization us-ing just stereo audio
and camera meta-data, without any visual input. Experimental results on a newly
collected Au-ditory Vehicle Tracking dataset verify that our proposed approach
outperforms several baseline approaches. We also demonstrate that our
cross-modal auditory localization approach can assist in the visual
localization of moving vehicles under poor lighting conditions.

该论文提出了一种利用未标记的音频 - 视觉数据进行物体定位的系统，使用自监督方法来实现跨模态的语音定位，其在声学车辆跟踪数据集上表现卓越并可用于光照条件不佳的车辆视觉定位。