Few-shot audio-visual acoustics modeling seeks to synthesize the room impulse
response in arbitrary locations with few-shot observations. To sufficiently
exploit the provided few-shot data for accurate acoustic modeling, we present a
*map-guided* framework by constructing acoustic-related visual semantic feature
maps of the scenes. Visual features preserve semantic details related to sound
and maps provide explicit structural regularities of sound propagation, which
are valuable for modeling environment acoustics. We thus extract pixel-wise
semantic features derived from observations and project them into a top-down
map, namely the **observation semantic map**. This map contains the relative
positional information among points and the semantic feature information
associated with each point. Yet, limited information extracted by few-shot
observations on the map is not sufficient for understanding and modeling the
whole scene. We address the challenge by generating a **scene semantic map**
via diffusing features and anticipating the observation semantic map. The scene
semantic map then interacts with echo encoding by a transformer-based
encoder-decoder to predict RIR for arbitrary speaker-listener query pairs.
Extensive experiments on Matterport3D and Replica dataset verify the efficacy
of our framework.

通过构造与声音相关的视觉语义特征图和地图，我们提出了一个以地图为导向的框架，通过少量的观测数据准确地建模环境声学。我们通过扩散特征和预测观测语义地图生成场景语义地图，然后通过基于 Transformer 的编码器 - 解码器与回声编码交互，预测任意发言者 - 听众查询对的房间冲激响应。大量在 Matterport3D 和 Replica 数据集上进行的实验验证了我们框架的有效性。