In this paper, we address the problem of enhancing the speech of a speaker of
interest in a cocktail party scenario when visual information of the speaker of
interest is available. Contrary to most previous studies, we do not learn
visual features on the typically small audio-visual datasets, but use an
already available face landmark detector (trained on a separate image dataset).
The landmarks are used by LSTM-based models to generate time-frequency masks
which are applied to the acoustic mixed-speech spectrogram. Results show that:
(i) landmark motion features are very effective features for this task, (ii)
similarly to previous work, reconstruction of the target speaker's spectrogram
mediated by masking is significantly more accurate than direct spectrogram
reconstruction, and (iii) the best masks depend on both motion landmark
features and the input mixed-speech spectrogram. To the best of our knowledge,
our proposed models are the first models trained and evaluated on the limited
size GRID and TCD-TIMIT datasets, that achieve speaker-independent speech
enhancement in a multi-talker setting.

本文提出了使用已有的人脸标记检测器生成时间 - 频率蒙版来提高多人交谈环境下的语音增强过程，其结果表明，我们的模型是在有限的 GRID 和 TCD-TIMIT 数据集上进行培训和评估的首个能够实现在多人交谈环境中实现独立于发言人的语音增强的模型。