One crucial challenge of real-world multilingual speech recognition is the
long-tailed distribution problem, where some resource-rich languages like
English have abundant training data, but a long tail of low-resource languages
have varying amounts of limited training data. To overcome the long-tail
problem, in this paper, we propose Adapt-and-Adjust (A2), a transformer-based
multi-task learning framework for end-to-end multilingual speech recognition.
The A2 framework overcomes the long-tail problem via three techniques: (1)
exploiting a pretrained multilingual language model (mBERT) to improve the
performance of low-resource languages; (2) proposing dual adapters consisting
of both language-specific and language-agnostic adaptation with minimal
additional parameters; and (3) overcoming the class imbalance, either by
imposing class priors in the loss during training or adjusting the logits of
the softmax output during inference. Extensive experiments on the CommonVoice
corpus show that A2 significantly outperforms conventional approaches.

本文提出了一种基于 Transformer 的多任务学习框架 Adapt-and-Adjust (A2)，通过利用预先训练好的多语种语言模型 (mBERT) 来提高低资源语言的性能、提出有着最小额外参数的语言特定和语言不可知适应器，以及在训练期间通过强制类先验或调整 softmax 输出的逻辑回归来克服类失衡。在 CommonVoice 数据集上的广泛实验表明，A2 明显优于传统方法。