We investigate the benefit of combining blind audio recordings with 3D scene
information for novel-view acoustic synthesis. Given audio recordings from 2-4
microphones and the 3D geometry and material of a scene containing multiple
unknown sound sources, we estimate the sound anywhere in the scene. We identify
the main challenges of novel-view acoustic synthesis as sound source
localization, separation, and dereverberation. While naively training an
end-to-end network fails to produce high-quality results, we show that
incorporating room impulse responses (RIRs) derived from 3D reconstructed rooms
enables the same network to jointly tackle these tasks. Our method outperforms
existing methods designed for the individual tasks, demonstrating its
effectiveness at utilizing 3D visual information. In a simulated study on the
Matterport3D-NVAS dataset, our model achieves near-perfect accuracy on source
localization, a PSNR of 26.44 dB and a SDR of 14.23 dB for source separation
and dereverberation, resulting in a PSNR of 25.55 dB and a SDR of 14.20 dB on
novel-view acoustic synthesis. Code, pretrained model, and video results are
available on the project webpage (this https URL).

探究结合盲音频录音和 3D 场景信息对新视角声学合成的益处。通过 2-4 个麦克风的音频录音和包含多个未知声源的场景的 3D 几何学和材料，估计场景中的任何声音。我们确定了新视角声学合成的主要挑战，即声源定位、分离和去混响。虽然简单地训练端到端网络不能产生高质量的结果，但我们证明了融入从 3D 重建房间导出的房间脉冲响应（RIRs）使得相同的网络能够统一解决这些任务。我们的方法优于针对各个任务设计的现有方法，证明其在利用 3D 视觉信息方面的有效性。在 Matterport3D-NVAS 数据集上的模拟研究中，我们的模型在源定位方面实现了接近完美的准确性，源分离和去混响方面的 PSNR 为 26.44 dB 和 SDR 为 14.23 dB，从而在新视角声学合成方面获得了 PSNR 为 25.55 dB 和 SDR 为 14.20 dB 的结果。项目网页提供了代码、预训练模型和视频结果。