This paper presents a streaming speaker-attributed automatic speech
recognition (SA-ASR) model that can recognize ``who spoke what'' with low
latency even when multiple people are speaking simultaneously. Our model is
based on token-level serialized output training (t-SOT) which was recently
proposed to transcribe multi-talker speech in a streaming fashion. To further
recognize speaker identities, we propose an encoder-decoder based speaker
embedding extractor that can estimate a speaker representation for each
recognized token not only from non-overlapping speech but also from overlapping
speech. The proposed speaker embedding, named t-vector, is extracted
synchronously with the t-SOT ASR model, enabling joint execution of speaker
identification (SID) or speaker diarization (SD) with the multi-talker
transcription with low latency. We evaluate the proposed model for a joint task
of ASR and SID/SD by using LibriSpeechMix and LibriCSS corpora. The proposed
model achieves substantially better accuracy than a prior streaming model and
shows comparable or sometimes even superior results to the state-of-the-art
offline SA-ASR model.

提出了一种基于 token 级别序列化输出训练 (t-SOT) 的流式说话者归属性自动语音识别 (SA-ASR) 模型，该模型可以在多人同时说话时实现低延迟的 “谁说了什么” 的识别，并提出了一种基于编码 - 解码的说话者嵌入提取器，可以从非重叠语音和重叠语音中为每个识别的令牌估计说话者表示。

使用令牌级别的说话人嵌入进行分配流式讲话者自动语音识别

Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

This paper proposes a token-level serialized output training (t-SOT), a novel
framework for streaming multi-talker automatic speech recognition (ASR). Unlike
existing streaming multi-talker ASR models using multiple output branches, the
t-SOT model has only a single output branch that generates recognition tokens
(e.g., words, subwords) of multiple speakers in chronological order based on
their emission times. A special token that indicates the change of ``virtual''
output channels is introduced to keep track of the overlapping utterances.
Compared to the prior streaming multi-talker ASR models, the t-SOT model has
the advantages of less inference cost and a simpler model architecture.
Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the
t-SOT-based transformer transducer model achieves the state-of-the-art word
error rates by a significant margin to the prior results. For non-overlapping
speech, the t-SOT model is on par with a single-talker ASR model in terms of
both accuracy and computational cost, opening the door for deploying one model
for both single- and multi-talker scenarios.

本文提出了一种基于 token 级别序列化输出训练（t-SOT）的新型流式多说话人自动语音识别框架，该框架在 LibriSpeechMix 和 LibriCSS 数据集中显示出比以前的结果更高的单词错误率的优势。