In this paper we present a unified time-frequency method for speaker
extraction in clean and noisy conditions. Given a mixed signal, along with a
reference signal, the common approaches for extracting the desired speaker are
either applied in the time-domain or in the frequency-domain. In our approach,
we propose a Siamese-Unet architecture that uses both representations. The
Siamese encoders are applied in the frequency-domain to infer the embedding of
the noisy and reference spectra, respectively. The concatenated representations
are then fed into the decoder to estimate the real and imaginary components of
the desired speaker, which are then inverse-transformed to the time-domain. The
model is trained with the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR)
loss to exploit the time-domain information. The time-domain loss is also
regularized with frequency-domain loss to preserve the speech patterns.
Experimental results demonstrate that the unified approach is not only very
easy to train, but also provides superior results as compared with
state-of-the-art (SOTA) Blind Source Separation (BSS) methods, as well as
commonly used speaker extraction approach.

本文提出了一种统一的时频方法，在清洁和嘈杂的条件下进行说话者提取。该方法采用了 Siamese-Unet 结构，使用频率域和时间域表示，以及采用了规范化的频域损失，最终在实验结果上达到了比传统的说话者提取方法和盲源分离方法更好的效果。