Developing embodied agents in simulation has been a key research topic in
recent years. Exciting new tasks, algorithms, and benchmarks have been
developed in various simulators. However, most of them assume deaf agents in
silent environments, while we humans perceive the world with multiple senses.
We introduce Sonicverse, a multisensory simulation platform with integrated
audio-visual simulation for training household agents that can both see and
hear. Sonicverse models realistic continuous audio rendering in 3D environments
in real-time. Together with a new audio-visual VR interface that allows humans
to interact with agents with audio, Sonicverse enables a series of embodied AI
tasks that need audio-visual perception. For semantic audio-visual navigation
in particular, we also propose a new multi-task learning model that achieves
state-of-the-art performance. In addition, we demonstrate Sonicverse's realism
via sim-to-real transfer, which has not been achieved by other simulators: an
agent trained in Sonicverse can successfully perform audio-visual navigation in
real-world environments. Sonicverse is available at:
this https URL

Sonicverse is a multisensory simulation platform for training household agents with audio-visual perception capabilities, incorporating a new audio-visual VR interface for human-agent interaction, and includes a multi-task learning model for audio-visual navigation, achieving state-of-the-art performance, and sim-to-real transfer capability.

Sonicverse: 面向具有视听能力的智能家居代理的多感官仿真平台

Sonicverse: A Multisensory Simulation Platform for Embodied Household  Agents that See and Hear

The ability to accurately recognize, localize and separate sound sources is
fundamental to any audio-visual perception task. Historically, these abilities
were tackled separately, with several methods developed independently for each
task. However, given the interconnected nature of source localization,
separation, and recognition, independent models are likely to yield suboptimal
performance as they fail to capture the interdependence between these tasks. To
address this problem, we propose a unified audio-visual learning framework
(dubbed OneAVM) that integrates audio and visual cues for joint localization,
separation, and recognition. OneAVM comprises a shared audio-visual encoder and
task-specific decoders trained with three objectives. The first objective
aligns audio and visual representations through a localized audio-visual
correspondence loss. The second tackles visual source separation using a
traditional mix-and-separate framework. Finally, the third objective reinforces
visual feature separation and localization by mixing images in pixel space and
aligning their representations with those of all corresponding sound sources.
Extensive experiments on MUSIC, VGG-Instruments, VGG-Music, and VGGSound
datasets demonstrate the effectiveness of OneAVM for all three tasks,
audio-visual source localization, separation, and nearest neighbor recognition,
and empirically demonstrate a strong positive transfer between them.

该研究提出了一种名为 OneAVM 的联合学习框架，该框架可以用于音频 - 视频源定位、分离和识别任务，其中共享的音频 - 视频编码器和任务特定的解码器是通过三个目标进行训练，包括本地化的音频 - 视觉对应丢失、视觉源分离和选择和用于强化视觉特征分离和定位的像素空间混合。经过在多个数据集上的广泛实验，证明了 OneAVM 的有效性，并在音频 - 视觉源定位、分离和最近邻识别任务之间展现了强大的正向转移。