Few-shot audio-visual acoustics modeling seeks to synthesize the room impulse
response in arbitrary locations with few-shot observations. To sufficiently
exploit the provided few-shot data for accurate acoustic modeling, we present a
*map-guided* framework by constructing acoustic-related visual semantic feature
maps of the scenes. Visual features preserve semantic details related to sound
and maps provide explicit structural regularities of sound propagation, which
are valuable for modeling environment acoustics. We thus extract pixel-wise
semantic features derived from observations and project them into a top-down
map, namely the **observation semantic map**. This map contains the relative
positional information among points and the semantic feature information
associated with each point. Yet, limited information extracted by few-shot
observations on the map is not sufficient for understanding and modeling the
whole scene. We address the challenge by generating a **scene semantic map**
via diffusing features and anticipating the observation semantic map. The scene
semantic map then interacts with echo encoding by a transformer-based
encoder-decoder to predict RIR for arbitrary speaker-listener query pairs.
Extensive experiments on Matterport3D and Replica dataset verify the efficacy
of our framework.

通过构造与声音相关的视觉语义特征图和地图，我们提出了一个以地图为导向的框架，通过少量的观测数据准确地建模环境声学。我们通过扩散特征和预测观测语义地图生成场景语义地图，然后通过基于 Transformer 的编码器 - 解码器与回声编码交互，预测任意发言者 - 听众查询对的房间冲激响应。大量在 Matterport3D 和 Replica 数据集上进行的实验验证了我们框架的有效性。

MAGIC: 地图指导的少样本音频 - 视觉声学建模

MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modeling

Visual object navigation using learning methods is one of the key tasks in
mobile robotics. This paper introduces a new representation of a scene semantic
map formed during the embodied agent interaction with the indoor environment.
It is based on a neural network method that adjusts the weights of the
segmentation model with backpropagation of the predicted fusion loss values
during inference on a regular (backward) or delayed (forward) image sequence.
We have implemented this representation into a full-fledged navigation approach
called SkillTron, which can select robot skills from end-to-end policies based
on reinforcement learning and classic map-based planning methods. The proposed
approach makes it possible to form both intermediate goals for robot
exploration and the final goal for object navigation. We conducted intensive
experiments with the proposed approach in the Habitat environment, which showed
a significant superiority in navigation quality metrics compared to
state-of-the-art approaches. The developed code and used custom datasets are
publicly available at github.com/AIRI-Institute/skill-fusion.

使用学习方法的视觉对象导航是移动机器人的关键任务之一，本文介绍了一种在与室内环境进行混合体互动过程中形成的场景语义地图的新表示方法，该方法基于神经网络通过反向或正向图像序列上的预测融合损失值的反向传播来调整分割模型的权重。我们将此表示方法实现到一种名为 SkillTron 的完整导航方法中，该方法可以根据强化学习和基于地图的经典规划方法从端到端策略中选择机器人技能。所提出的方法能够为机器人探索制定中间目标和对象导航制定最终目标，我们在 Habitat 环境中对所提出的方法进行了大量实验，在导航质量指标方面表现出明显的优势，与现有技术方法相比。所开发的代码和使用的自定义数据集可以在 github.com/AIRI-Institute/skill-fusion 上公开获取。