Recent improvements in neural audio codec (NAC) models have generated
interest in adopting pre-trained codecs for a variety of speech processing
applications to take advantage of the efficiencies gained from high
compression, but these have yet been applied to the speech separation (SS)
task. SS can benefit from high compression because the compute required for
traditional SS models makes them impractical for many edge computing use cases.
However, SS is a waveform-masking task where compression tends to introduce
distortions that severely impact performance. Here we propose a novel task of
Audio Codec-based SS, where SS is performed within the embedding space of a
NAC, and propose a new model, Codecformer, to address this task. At inference,
Codecformer achieves a 52x reduction in MAC while producing separation
performance comparable to a cloud deployment of Sepformer. This method charts a
new direction for performing efficient SS in practical scenarios.

通过在 NAC 的嵌入空间中进行音频编码器（Codec）的语音分离（SS）任务，我们提出了一种新的模型 Codecformer，在推断过程中实现了 52 倍的 MAC 降低，并且产生了与 Sepformer 云部署相当的分离性能，为在实际场景中实现高效的 SS 打开了新的方向。