This paper proposes a token-level serialized output training (t-SOT), a novel
framework for streaming multi-talker automatic speech recognition (ASR). Unlike
existing streaming multi-talker ASR models using multiple output branches, the
t-SOT model has only a single output branch that generates recognition tokens
(e.g., words, subwords) of multiple speakers in chronological order based on
their emission times. A special token that indicates the change of ``virtual''
output channels is introduced to keep track of the overlapping utterances.
Compared to the prior streaming multi-talker ASR models, the t-SOT model has
the advantages of less inference cost and a simpler model architecture.
Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the
t-SOT-based transformer transducer model achieves the state-of-the-art word
error rates by a significant margin to the prior results. For non-overlapping
speech, the t-SOT model is on par with a single-talker ASR model in terms of
both accuracy and computational cost, opening the door for deploying one model
for both single- and multi-talker scenarios.

本文提出了一种基于 token 级别序列化输出训练（t-SOT）的新型流式多说话人自动语音识别框架，该框架在 LibriSpeechMix 和 LibriCSS 数据集中显示出比以前的结果更高的单词错误率的优势。