In this paper, we focus on addressing the constraints faced when applying
LLMs to ASR. Recent works utilize prefixLM-type models, which directly apply
speech as a prefix to LLMs for ASR. We have found that optimizing speech
prefixes leads to better ASR performance and propose applying RNNT loss to
perform speech prefix-tuning. This is a simple approach and does not increase
the model complexity or alter the inference pipeline. We also propose
language-based soft prompting to further improve with frozen LLMs. Empirical
analysis on realtime testset from 10 Indic languages demonstrate that our
proposed speech prefix-tuning yields improvements with both frozen and
fine-tuned LLMs. Our recognition results on an average of 10 Indics show that
the proposed prefix-tuning with RNNT loss results in a 12\% relative
improvement in WER over the baseline with a fine-tuned LLM. Our proposed
approches with the frozen LLM leads to a 31\% relative improvement over basic
soft-prompting prefixLM.

本研究针对将 LLMs 应用于 ASR 时面临的限制进行了讨论，提出了使用 RNNT 损失进行语音前缀调优的方法，并通过语言软提示进一步改善，实证分析表明我们的方法在多种语言上取得了显著提升。