In real-world applications, users often require both translations and
transcriptions of speech to enhance their comprehension, particularly in
streaming scenarios where incremental generation is necessary. This paper
introduces a streaming Transformer-Transducer that jointly generates automatic
speech recognition (ASR) and speech translation (ST) outputs using a single
decoder. To produce ASR and ST content effectively with minimal latency, we
propose a joint token-level serialized output training method that interleaves
source and target words by leveraging an off-the-shelf textual aligner.
Experiments in monolingual (it-en) and multilingual (\{de,es,it\}-en) settings
demonstrate that our approach achieves the best quality-latency balance. With
an average ASR latency of 1s and ST latency of 1.3s, our model shows no
degradation or even improves output quality compared to separate ASR and ST
models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the
multilingual case.

该论文介绍了一种使用单个解码器共同生成自动语音识别和语音翻译输出的流式 Transformer-Transducer，并提出了一种联合令牌级串行输出训练方法以实现有效的 ASR 和 ST 内容生成，其平均 ASR 延迟为 1 秒，ST 延迟为 1.3 秒，在多语言情况下优化了输出质量表现。