We develop a large language model (LLM) based automatic speech recognition (ASR) system that can be contextualized by providing keywords as prior information in text prompts. We adopt decoder-only architecture and use our in-house LLM, PLaMo-100B, pre-trained from scratch using datasets dominated by Japanese and English texts as the decoder. We adopt a pre-trained Whisper encoder as an audio encoder, and the audio embeddings from the audio encoder are projected to the text embedding space by an adapter layer and concatenated with text embeddings converted from text prompts to form inputs to the decoder. By providing keywords as prior information in the text prompts, we can contextualize our LLM-based ASR system without modifying the model architecture to transcribe ambiguous words in the input audio accurately. Experimental results demonstrate that providing keywords to the decoder can significantly improve the recognition performance of rare and ambiguous words.

本研究解决了语音识别系统在处理稀有和模糊词汇时的准确性不足问题。通过在文本提示中提供关键词作为先前信息，作者提出了一种新的自回归解码架构，能够在不修改模型架构的情况下对模糊词进行准确转录。实验结果表明，该方法显著提高了罕见和模糊词汇的识别性能。

通过上下文化提高基于大语言模型的语音识别能力以识别稀有和模糊词汇