Speaker extraction aims to mimic humans' selective auditory attention by
extracting a target speaker's voice from a multi-talker environment. It is
common to perform the extraction in frequency-domain, and reconstruct the
time-domain signal from the extracted magnitude and estimated phase spectra.
However, such an approach is adversely affected by the inherent difficulty of
phase estimation. Inspired by Conv-TasNet, we propose a time-domain speaker
extraction network (SpEx) that converts the mixture speech into multi-scale
embedding coefficients instead of decomposing the speech signal into magnitude
and phase spectra. In this way, we avoid phase estimation. The SpEx network
consists of four network components, namely speaker encoder, speech encoder,
speaker extractor, and speech decoder. Specifically, the speech encoder
converts the mixture speech into multi-scale embedding coefficients, the
speaker encoder learns to represent the target speaker with a speaker
embedding. The speaker extractor takes the multi-scale embedding coefficients
and target speaker embedding as input and estimates a receptive mask. Finally,
the speech decoder reconstructs the target speaker's speech from the masked
embedding coefficients. We also propose a multi-task learning framework and a
multi-scale embedding implementation. Experimental results show that the
proposed SpEx achieves 37.3%, 37.7% and 15.0% relative improvements over the
best baseline in terms of signal-to-distortion ratio (SDR), scale-invariant SDR
(SI-SDR), and perceptual evaluation of speech quality (PESQ) under an open
evaluation condition.

通过提出 SpEx 网络并采用多尺度嵌入系数，将多说话声音信号转换成特定发言人的声音信号，该网络在信号畸变比（SDR）、比例尺不变信号畸变比（SI-SDR）和语音质量评估（PESQ）等方面均有显著改进。