The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audio-visual speech separation network. Unlike previous works that used lip movement on video clips or pre-enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker. In this task, the conditional feature is obtained from facial appearance in cross-modal biometric task, where audio and visual identity representations are shared in latent space. Learnt identities from facial images enforce the network to isolate matched speakers and extract the voices from mixed speech. It solves the permutation problem caused by swapped channel outputs, frequently occurred in speech separation tasks. The proposed method is far more practical than video-based speech separation since user profile images are readily available on many platforms. Also, unlike speaker-aware separation methods, it is applicable on separation with unseen speakers who have never been enrolled before. We show strong qualitative and quantitative results on challenging real-world examples.

使用深度学习中的音频-视觉语音分离网络，从混音状态中分离出指定说话者的语音，且仅使用目标说话者的单一面部图像，通过潜在空间的交叉模态生物识别任务中的面部外观获得条件特征，可隔离和提取这些说话者的语音，为语音分离任务中的通道交换问题提供解决方案，同时也适用于未知说话者。

FaceFilter: 通过静态图像进行音视频语音分离