Sound plays a major role in human perception, providing essential scene
information alongside vision for understanding our environment. Despite
progress in neural implicit representations, learning acoustics that match a
visual scene is still challenging. We propose NeRAF, a method that jointly
learns acoustic and radiance fields. NeRAF is designed as a Nerfstudio module
for convenient access to realistic audio-visual generation. It synthesizes both
novel views and spatialized audio at new positions, leveraging radiance field
capabilities to condition the acoustic field with 3D scene information. At
inference, each modality can be rendered independently and at spatially
separated positions, providing greater versatility. We demonstrate the
advantages of our method on the SoundSpaces dataset. NeRAF achieves substantial
performance improvements over previous works while being more data-efficient.
Furthermore, NeRAF enhances novel view synthesis of complex scenes trained with
sparse data through cross-modal learning.

NeRAF 是一种同时学习声音和辐射场的方法，用于实现逼真的视听生成，通过 3D 场景信息将声学场与辐射场联系起来。在 SoundSpaces 数据集上，NeRAF 相比之前的方法在性能上有显著的改进，并且具有更高的数据效率，同时通过跨模态学习增强了稀疏数据训练的复杂场景的新视图合成。