The objective of this work is to extract target speaker's voice from a
mixture of voices using visual cues. Existing works on audio-visual speech
separation have demonstrated their performance with promising intelligibility,
but maintaining naturalness remains a challenge. To address this issue, we
propose AVDiffuSS, an audio-visual speech separation model based on a diffusion
mechanism known for its capability in generating natural samples. For an
effective fusion of the two modalities for diffusion, we also propose a
cross-attention-based feature fusion mechanism. This mechanism is specifically
tailored for the speech domain to integrate the phonetic information from
audio-visual correspondence in speech generation. In this way, the fusion
process maintains the high temporal resolution of the features, without
excessive computational requirements. We demonstrate that the proposed
framework achieves state-of-the-art results on two benchmarks, including
VoxCeleb2 and LRS3, producing speech with notably better naturalness.

本文介绍了 AVDiffuSS，一种基于扩散机制的音视频语音分离模型，通过视觉线索从声音混合中提取目标发言者的声音，该模型在保持自然性方面具有挑战，并提出了一种基于交叉注意力的特征融合机制，以实现两种模态的有效融合，并在语音生成中集成语音视觉对应的语音信息，通过该提出的框架在 VoxCeleb2 和 LRS3 这两个基准测试上取得了最先进的结果，生成的语音具有显著更好的自然音质。