The growing need for instant spoken language transcription and translation is
driven by increased global communication and cross-lingual interactions. This
has made offering translations in multiple languages essential for user
applications. Traditional approaches to automatic speech recognition (ASR) and
speech translation (ST) have often relied on separate systems, leading to
inefficiencies in computational resources, and increased synchronization
complexity in real time. In this paper, we propose a streaming
Transformer-Transducer (T-T) model able to jointly produce many-to-one and
one-to-many transcription and translation using a single decoder. We introduce
a novel method for joint token-level serialized output training based on
timestamp information to effectively produce ASR and ST outputs in the
streaming setting. Experiments on {it,es,de}->en prove the effectiveness of our
approach, enabling the generation of one-to-many joint outputs with a single
decoder for the first time.

提出了一种流式 Transformer-Transducer (T-T) 模型，能够使用单个解码器联合生成多对一和一对多的转录和翻译，并引入了一种基于时间戳信息的新颖方法来有效地在流式环境中生成语音识别和语音翻译的输出。通过在 {it, es, de}->en 上进行的实验证明了我们方法的有效性，首次实现了使用单个解码器生成一对多联合输出。