While large language models excel in a variety of natural language processing
(NLP) tasks, to perform well on spoken language understanding (SLU) tasks, they
must either rely on off-the-shelf automatic speech recognition (ASR) systems
for transcription, or be equipped with an in-built speech modality. This work
focuses on the former scenario, where LLM's accuracy on SLU tasks is
constrained by the accuracy of a fixed ASR system on the spoken input.
Specifically, we tackle speech-intent classification task, where a high
word-error-rate can limit the LLM's ability to understand the spoken intent.
Instead of chasing a high accuracy by designing complex or specialized
architectures regardless of deployment costs, we seek to answer how far we can
go without substantially changing the underlying ASR and LLM, which can
potentially be shared by multiple unrelated tasks. To this end, we propose
prompting the LLM with an n-best list of ASR hypotheses instead of only the
error-prone 1-best hypothesis. We explore prompt-engineering to explain the
concept of n-best lists to the LLM; followed by the finetuning of Low-Rank
Adapters on the downstream tasks. Our approach using n-best lists proves to be
effective on a device-directed speech detection task as well as on a keyword
spotting task, where systems using n-best list prompts outperform those using
1-best ASR hypothesis; thus paving the way for an efficient method to exploit
ASR uncertainty via LLMs for speech-based applications.

利用 n-best 列表提示方法，我们改善了基于大型语言模型的口语理解任务，使其能够更好地理解口语意图并应用于基于语音的应用。