End-to-end formulation of automatic speech recognition (ASR) and speech
translation (ST) makes it easy to use a single model for both multilingual ASR
and many-to-many ST. In this paper, we propose streaming language-agnostic
multilingual speech recognition and translation using neural transducers
(LAMASSU). To enable multilingual text generation in LAMASSU, we conduct a
systematic comparison between specified and unified prediction and joint
networks. We leverage a language-agnostic multilingual encoder that
substantially outperforms shared encoders. To enhance LAMASSU, we propose to
feed target LID to encoders. We also apply connectionist temporal
classification regularization to transducer training. Experimental results show
that LAMASSU not only drastically reduces the model size but also outperforms
monolingual ASR and bilingual ST models.

本文提出了一种使用神经转导器的流式语言无关的多语言语音识别和翻译模型，采用语言无关的多语言编码器显著提高了模型效果，并探究统一预测和联合网络的优缺点，通过给编码器输入目标 LID 和连接时序分类正则化的训练，不仅显著降低了模型大小，还取得了优于单语音识别和双语言翻译模型的结果。