Can conversational videos captured from multiple egocentric viewpoints reveal
the map of a scene in a cost-efficient way? We seek to answer this question by
proposing a new problem: efficiently building the map of a previously unseen 3D
environment by exploiting shared information in the egocentric audio-visual
observations of participants in a natural conversation. Our hypothesis is that
as multiple people ("egos") move in a scene and talk among themselves, they
receive rich audio-visual cues that can help uncover the unseen areas of the
scene. Given the high cost of continuously processing egocentric visual
streams, we further explore how to actively coordinate the sampling of visual
information, so as to minimize redundancy and reduce power use. To that end, we
present an audio-visual deep reinforcement learning approach that works with
our shared scene mapper to selectively turn on the camera to efficiently chart
out the space. We evaluate the approach using a state-of-the-art audio-visual
simulator for 3D scenes as well as real-world video. Our model outperforms
previous state-of-the-art mapping methods, and achieves an excellent
cost-accuracy tradeoff. Project: this http URL

利用多个自我视角的、来自多方的对话视频，结合深度强化学习的方法，高效地构建出未知 3D 环境的地图，达到了出色的性价比，优于以往最先进的方法。