We present TokenSplit, a speech separation model that acts on discrete token
sequences. The model is trained on multiple tasks simultaneously: separate and
transcribe each speech source, and generate speech from text. The model
operates on transcripts and audio token sequences and achieves multiple tasks
through masking of inputs. The model is a sequence-to-sequence encoder-decoder
model that uses the Transformer architecture. We also present a "refinement"
version of the model that predicts enhanced audio tokens from the audio tokens
of speech separated by a conventional separation model. Using both objective
metrics and subjective MUSHRA listening tests, we show that our model achieves
excellent performance in terms of separation, both with or without transcript
conditioning. We also measure the automatic speech recognition (ASR)
performance and provide audio samples of speech synthesis to demonstrate the
additional utility of our model.

我们提出了 TokenSplit，一种在离散标记序列上操作的语音分离模型。该模型通过输入掩码实现多任务训练，包括将每个语音源进行分离和转录，并从文本生成语音。我们还介绍了模型的 “修正” 版本，该版本通过传统分离模型分离的音频标记预测增强音频标记。通过客观度量和主观 MUSHRA 听测，我们证明了我们的模型在分离方面具有优秀的性能，无论是否有转录条件。我们还测量了自动语音识别（ASR）性能，并提供了语音合成的音频样本来展示我们模型的附加效用。