Despite the rapid progress in automatic speech recognition (ASR) research,
recognizing multilingual speech using a unified ASR system remains highly
challenging. Previous works on multilingual speech recognition mainly focus on
two directions: recognizing multiple monolingual speech or recognizing
code-switched speech that uses different languages interchangeably within a
single utterance. However, a pragmatic multilingual recognizer is expected to
be compatible with both directions. In this work, a novel language-aware
encoder (LAE) architecture is proposed to handle both situations by
disentangling language-specific information and generating frame-level
language-aware representations during encoding. In the LAE, the primary
encoding is implemented by the shared block while the language-specific blocks
are used to extract specific representations for each language. To learn
language-specific information discriminatively, a language-aware training
method is proposed to optimize the language-specific blocks in LAE. Experiments
conducted on Mandarin-English code-switched speech suggest that the proposed
LAE is capable of discriminating different languages in frame-level and shows
superior performance on both monolingual and multilingual ASR tasks. With
either a real-recorded or simulated code-switched dataset, the proposed LAE
achieves statistically significant improvements on both CTC and neural
transducer systems. Code is released

该研究提出了一种新的语言感知编码器（LAE）架构，通过解开语言特定信息和在编码期间生成帧级语言感知表示来处理多语言语音识别，并且在华语 - 英语混合语音上表现出卓越的性能。