Accurate estimation of Room Impulse Response (RIR), which captures an
environment's acoustic properties, is important for speech processing and AR/VR
applications. We propose AV-RIR, a novel multi-modal multi-task learning
approach to accurately estimate the RIR from a given reverberant speech signal
and the visual cues of its corresponding environment. AV-RIR builds on a novel
neural codec-based architecture that effectively captures environment geometry
and materials properties and solves speech dereverberation as an auxiliary task
by using multi-task learning. We also propose Geo-Mat features that augment
material information into visual cues and CRIP that improves late reverberation
components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical
results show that AV-RIR quantitatively outperforms previous audio-only and
visual-only approaches by achieving 36% - 63% improvement across various
acoustic metrics in RIR estimation. Additionally, it also achieves higher
preference scores in human evaluation. As an auxiliary benefit, dereverbed
speech from AV-RIR shows competitive performance with the state-of-the-art in
various spoken language processing tasks and outperforms reverberation time
error score in the real-world AVSpeech dataset. Qualitative examples of both
synthesized reverberant speech and enhanced speech can be found at
this https URL

准确估计室内脉冲响应（Room Impulse Response，RIR）对于语音处理和增强实境 / 虚境应用非常重要。本文提出了 AV-RIR，一种新颖的多模态多任务学习方法，能够准确从给定的回声语音信号和对应环境的视觉线索中估计 RIR。AV-RIR 基于一种能有效捕捉环境几何和材质特性的新型神经编解码器架构，通过多任务学习将语音降噪作为辅助任务进行求解。我们还提出了 Geo-Mat 特征，将材料信息加入到视觉线索中，并通过图像 - RIR 检索改进了所估计的 RIR 中的后期混响成分。实证结果表明，AV-RIR 在 RIR 估计的各种声学度量标准上优于先前的仅音频和仅视觉方法，改进幅度在 36% 至 63% 之间。此外，它在人类评估中也获得了更高的偏好评分。作为一个附带效果，AV-RIR 的去混响语音在各种口语处理任务中显示出竞争性能，并在实际 AVSpeech 数据集中超过了混响时间误差评分。可以在此 https URL 找到合成的回声语音和增强后的语音的定性示例。