Moving around in the world is naturally a multisensory experience, but
today's embodied agents are deaf---restricted to solely their visual perception
of the environment. We introduce audio-visual navigation for complex,
acoustically and visually realistic 3D environments. By both seeing and
hearing, the agent must learn to navigate to a sounding object. We propose a
multi-modal deep reinforcement learning approach to train navigation policies
end-to-end from a stream of egocentric audio-visual observations, allowing the
agent to (1) discover elements of the geometry of the physical space indicated
by the reverberating audio and (2) detect and follow sound-emitting targets. We
further introduce SoundSpaces: a first-of-its-kind dataset of audio renderings
based on geometrical acoustic simulations for two sets of publicly available 3D
environments (Matterport3D and Replica), and we instrument Habitat to support
the new sensor, making it possible to insert arbitrary sound sources in an
array of real-world scanned environments. Our results show that audio greatly
benefits embodied visual navigation in 3D spaces, and our work lays groundwork
for new research in embodied AI with audio-visual perception.

本研究介绍了一种复杂的，声音和视觉逼真的三维环境中的音频视觉导航方法，使用多模态深度强化学习方法训练导航策略并提出了新的数据集 SoundSpaces，在现实环境中插入任意声源。研究结果表明，在三维空间中，音频对于具身视觉导航有很大的帮助，为音频视觉感知的体验机器人研究奠定了基础。