The utilization of speech Self-Supervised Learning (SSL) models achieves impressive performance on Automatic Speech Recognition (ASR). However, in low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. Typical solutions like fine-tuning the SSL model suffer from high computation costs while using frozen SSL models as feature extractors comes with poor performance. To handle these issues, we extend a conventional efficient fine-tuning scheme based on the adapter. We add an extra intermediate adaptation to warm up the adapter and downstream model initialization. Remarkably, we update only 1-5% of the total model parameters to achieve the adaptation. Experimental results on the ML-SUPERB dataset show that our solution outperforms conventional efficient fine-tuning. It achieves up to a 28% relative improvement in the Character/Phoneme error rate when adapting to unseen languages.

本研究针对自监督学习模型在低资源语言自动语音识别中的领域差异问题，提出了一种改进的高效微调方案。通过增加中间适应步骤，仅更新1-5%的模型参数，实验证明该方法在适应未见语言时，相较传统微调方法提高了多达28%的字符和音素错误率表现。

如何学习一门新语言？低资源场景中自监督学习模型适应未见语言的高效解决方案