Speaker extraction seeks to extract the target speech in a multi-talker
scenario given an auxiliary reference. Such reference can be auditory, i.e., a
pre-recorded speech, visual, i.e., lip movements, or contextual, i.e., phonetic
sequence. References in different modalities provide distinct and complementary
information that could be fused to form top-down attention on the target
speaker. Previous studies have introduced visual and contextual modalities in a
single model. In this paper, we propose a two-stage time-domain
visual-contextual speaker extraction network named VCSE, which incorporates
visual and self-enrolled contextual cues stage by stage to take full advantage
of every modality. In the first stage, we pre-extract a target speech with
visual cues and estimate the underlying phonetic sequence. In the second stage,
we refine the pre-extracted target speech with the self-enrolled contextual
cues. Experimental results on the real-world Lip Reading Sentences 3 (LRS3)
database demonstrate that our proposed VCSE network consistently outperforms
other state-of-the-art baselines.

本文提出了一种包含视觉和自我学习语境线索的两阶段时域视觉 —— 语境扬声器提取网络 VCSE, 在 LRS3 数据库上表现比其他最先进的基线一致优秀