Large language models (LLM) have demonstrated the ability to understand human language by leveraging large amount of text data. Automatic speech recognition (ASR) systems are often limited by available transcribed speech data and benefit from a second pass rescoring using LLM. Recently multi-modal large language models, particularly speech and text foundational models have demonstrated strong spoken language understanding. Speech-Text foundational models leverage large amounts of unlabelled and labelled data both in speech and text modalities to model human language. In this work, we propose novel techniques to use multi-modal LLM for ASR rescoring. We also explore discriminative training to further improve the foundational model rescoring performance. We demonstrate cross-modal knowledge transfer in speech-text LLM can benefit rescoring. Our experiments demonstrate up-to 20% relative improvements over Whisper large ASR and up-to 15% relative improvements over text-only LLM.

本研究解决了自动语音识别（ASR）系统因可用转录语音数据有限而导致的性能瓶颈。提出了利用多模态大型语言模型进行ASR重评分的创新技术，并探索了判别训练来提升重评分性能。实验表明，通过语音-文本基础模型的跨模态知识转移，重评分效果显著提升，相较Whisper大型ASR和仅基于文本的LLM分别提高了20%和15%的相对性能。 

基于大型语音-文本基础模型的语音识别重评分