Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively.

本文介绍了一种新颖的双语端到端模型，通过共享编码器和预测网络，并通过自注意机制组合语言特定的联合网络，实现了单一神经模型对多种语言的识别和支持语言之间的动态切换，相比传统双语基线模型，该方法在印地语、英语和混合代码测试集上分别降低了13.3%、8.23%和1.3%的词误差率。

使用多个softmax上的注意力的流式双语端到端自动语音识别模型