In audio-visual navigation, an agent intelligently travels through a complex,
unmapped 3D environment using both sights and sounds to find a sound source
(e.g., a phone ringing in another room). Existing models learn to act at a
fixed granularity of agent motion and rely on simple recurrent aggregations of
the audio observations. We introduce a reinforcement learning approach to
audio-visual navigation with two key novel elements: 1) waypoints that are
dynamically set and learned end-to-end within the navigation policy, and 2) an
acoustic memory that provides a structured, spatially grounded record of what
the agent has heard as it moves. Both new ideas capitalize on the synergy of
audio and visual data for revealing the geometry of an unmapped space. We
demonstrate our approach on two challenging datasets of real-world 3D scenes,
Replica and Matterport3D. Our model improves the state of the art by a
substantial margin, and our experiments reveal that learning the links between
sights, sounds, and space is essential for audio-visual navigation. Project:
this http URL

本文介绍了一种基于强化学习的音视觉导航方法，通过动态设置和学习的航点和声音记忆，利用视觉和声音数据揭示了未映射空间的几何结构，实验结果表明，学习视听空间之间的联系对于音视觉导航至关重要。