Accurate estimation of Room Impulse Response (RIR), which captures an
environment's acoustic properties, is important for speech processing and AR/VR
applications. We propose AV-RIR, a novel multi-modal multi-task learning
approach to accurately estimate the RIR from a given reverberant speech signal
and the visual cues of its corresponding environment. AV-RIR builds on a novel
neural codec-based architecture that effectively captures environment geometry
and materials properties and solves speech dereverberation as an auxiliary task
by using multi-task learning. We also propose Geo-Mat features that augment
material information into visual cues and CRIP that improves late reverberation
components in the estimated RIR via image-to-RIR retrieval by 86%. Empirical
results show that AV-RIR quantitatively outperforms previous audio-only and
visual-only approaches by achieving 36% - 63% improvement across various
acoustic metrics in RIR estimation. Additionally, it also achieves higher
preference scores in human evaluation. As an auxiliary benefit, dereverbed
speech from AV-RIR shows competitive performance with the state-of-the-art in
various spoken language processing tasks and outperforms reverberation time
error score in the real-world AVSpeech dataset. Qualitative examples of both
synthesized reverberant speech and enhanced speech can be found at
this https URL

准确估计室内脉冲响应（Room Impulse Response，RIR）对于语音处理和增强实境 / 虚境应用非常重要。本文提出了 AV-RIR，一种新颖的多模态多任务学习方法，能够准确从给定的回声语音信号和对应环境的视觉线索中估计 RIR。AV-RIR 基于一种能有效捕捉环境几何和材质特性的新型神经编解码器架构，通过多任务学习将语音降噪作为辅助任务进行求解。我们还提出了 Geo-Mat 特征，将材料信息加入到视觉线索中，并通过图像 - RIR 检索改进了所估计的 RIR 中的后期混响成分。实证结果表明，AV-RIR 在 RIR 估计的各种声学度量标准上优于先前的仅音频和仅视觉方法，改进幅度在 36% 至 63% 之间。此外，它在人类评估中也获得了更高的偏好评分。作为一个附带效果，AV-RIR 的去混响语音在各种口语处理任务中显示出竞争性能，并在实际 AVSpeech 数据集中超过了混响时间误差评分。可以在此 https URL 找到合成的回声语音和增强后的语音的定性示例。

AV-RIR：音频 - 视觉房间冲激响应估计

AV-RIR: Audio-Visual Room Impulse Response Estimation

Speech applications in far-field real world settings often deal with signals
that are corrupted by reverberation. The task of dereverberation constitutes an
important step to improve the audible quality and to reduce the error rates in
applications like automatic speech recognition (ASR). We propose a unified
framework of speech dereverberation for improving the speech quality and the
ASR performance using the approach of envelope-carrier decomposition provided
by an autoregressive (AR) model. The AR model is applied in the frequency
domain of the sub-band speech signals to separate the envelope and carrier
parts. A novel neural architecture based on dual path long short term memory
(DPLSTM) model is proposed, which jointly enhances the sub-band envelope and
carrier components. The dereverberated envelope-carrier signals are modulated
and the sub-band signals are synthesized to reconstruct the audio signal back.
The DPLSTM model for dereverberation of envelope and carrier components also
allows the joint learning of the network weights for the down stream ASR task.
In the ASR tasks on the REVERB challenge dataset as well as on the VOiCES
dataset, we illustrate that the joint learning of speech dereverberation
network and the E2E ASR model yields significant performance improvements over
the baseline ASR system trained on log-mel spectrogram as well as other
benchmarks for dereverberation (average relative improvements of 10-24% over
the baseline system). The speech quality improvements, evaluated using
subjective listening tests, further highlight the improved quality of the
reconstructed audio.

通过自回归（AR）模型在子带语音信号的频域中分离包络和载波部分，提出了一种改善语音质量和 ASR 性能的语音去混响统一框架，设计了基于双路径长短期记忆（DPLSTM）模型的神经网络结构，实现改善了子带包络和载波组件，进一步通过联合学习网络权重来优化 ASR 性能，该方法在 REVERB 挑战数据集和 VOiCES 数据集上相较基线系统以及其他去混响基准方法显示出显著的性能改善（基线系统上相对提升率为 10-24%），主观听测试进一步突显了音频还原质量的提升。

基于频域自回归建模的语音增强

Speech enhancement with frequency domain auto-regressive modeling

We investigate the use of generative adversarial networks (GANs) in speech
dereverberation for robust speech recognition. GANs have been recently studied
for speech enhancement to remove additive noises, but there still lacks of a
work to examine their ability in speech dereverberation and the advantages of
using GANs have not been fully established. In this paper, we provide deep
investigations in the use of GAN-based dereverberation front-end in ASR. First,
we study the effectiveness of different dereverberation networks (the generator
in GAN) and find that LSTM leads a significant improvement as compared with
feed-forward DNN and CNN in our dataset. Second, further adding residual
connections in the deep LSTMs can boost the performance as well. Finally, we
find that, for the success of GAN, it is important to update the generator and
the discriminator using the same mini-batch data during training. Moreover,
using reverberant spectrogram as a condition to discriminator, as suggested in
previous studies, may degrade the performance. In summary, our GAN-based
dereverberation front-end achieves 14%-19% relative CER reduction as compared
to the baseline DNN dereverberation network when tested on a strong
multi-condition training acoustic model.

本文探讨了生成式对抗网络在语音去混响中的应用及其提高语音识别性能的优势，通过实验发现将 LSTM 与深度残差网络相结合可达到 16% 的语音识别误差率相对下降。