Speech enhancement systems are typically trained using pairs of clean and
noisy speech. In audio-visual speech enhancement (AVSE), there is not as much
ground-truth clean data available; most audio-visual datasets are collected in
real-world environments with background noise and reverberation, hampering the
development of AVSE. In this work, we introduce AV2Wav, a resynthesis-based
audio-visual speech enhancement approach that can generate clean speech despite
the challenges of real-world training data. We obtain a subset of nearly clean
speech from an audio-visual corpus using a neural quality estimator, and then
train a diffusion model on this subset to generate waveforms conditioned on
continuous speech representations from AV-HuBERT with noise-robust training. We
use continuous rather than discrete representations to retain prosody and
speaker information. With this vocoding task alone, the model can perform
speech enhancement better than a masking-based baseline. We further fine-tune
the diffusion model on clean/noisy utterance pairs to improve the performance.
Our approach outperforms a masking-based baseline in terms of both automatic
metrics and a human listening test and is close in quality to the target speech
in the listening test. Audio samples can be found at
this https URL

在这项工作中，我们介绍了 AV2Wav，这是一种基于重新综合的视听语音增强方法，可以在现实训练数据的挑战下生成清晰的语音。我们使用神经质量评估器从视听语音语料库中获取接近干净的子集，然后在这个子集上训练扩散模型，该模型以 AV-HuBERT 的连续语音表示为条件生成波形，并使用抗噪训练。我们使用连续表示而不是离散表示来保留韵律和说话人信息。仅仅通过这个声码任务，该模型的语音增强性能就优于基于掩模的基线。我们进一步在清晰 / 噪声话语对上对扩散模型进行微调以提高性能。我们的方法在自动度量和人工听力测试中均优于基于掩模的基线，并且在听力测试中的质量接近目标语音。