We introduce the active audio-visual source separation problem, where an
agent must move intelligently in order to better isolate the sounds coming from
an object of interest in its environment. The agent hears multiple audio
sources simultaneously (e.g., a person speaking down the hall in a noisy
household) and it must use its eyes and ears to automatically separate out the
sounds originating from a target object within a limited time budget. Towards
this goal, we introduce a reinforcement learning approach that trains movement
policies controlling the agent's camera and microphone placement over time,
guided by the improvement in predicted audio separation quality. We demonstrate
our approach in scenarios motivated by both augmented reality (system is
already co-located with the target object) and mobile robotics (agent begins
arbitrarily far from the target object). Using state-of-the-art realistic
audio-visual simulations in 3D environments, we demonstrate our model's ability
to find minimal movement sequences with maximal payoff for audio source
separation. Project: this http URL

本文介绍了一种使用增强学习来控制摄像头和麦克风位置提高音频分离质量的方法，解决了活跃的音视频源分离问题，并针对增强现实和移动机器人等场景进行了验证。