In the realm of spoken language understanding (SLU), numerous natural
language understanding (NLU) methodologies have been adapted by supplying large
language models (LLMs) with transcribed speech instead of conventional written
text. In real-world scenarios, prior to input into an LLM, an automated speech
recognition (ASR) system generates an output transcript hypothesis, where
inherent errors can degrade subsequent SLU tasks. Here we introduce a method
that utilizes the ASR system's lattice output instead of relying solely on the
top hypothesis, aiming to encapsulate speech ambiguities and enhance SLU
outcomes. Our in-context learning experiments, covering spoken question
answering and intent classification, underline the LLM's resilience to noisy
speech transcripts with the help of word confusion networks from lattices,
bridging the SLU performance gap between using the top ASR hypothesis and an
oracle upper bound. Additionally, we delve into the LLM's robustness to varying
ASR performance conditions and scrutinize the aspects of in-context learning
which prove the most influential.

通过使用自动语音识别系统的网络输出来弥补语音模糊性，增强口语语言理解的结果，通过对置信网络的词语混淆进行实验，研究大型语言模型对嘈杂的语音转录本的鲁棒性，探讨在不同自动语音识别性能条件下的影响因素。

通过上下文学习和词混淆网络实现 ASR 鲁棒性口语语言理解

Towards ASR Robust Spoken Language Understanding Through In-Context  Learning With Word Confusion Networks

Spoken Language Understanding (SLU) converts hypotheses from automatic speech
recognizer (ASR) into structured semantic representations. ASR recognition
errors can severely degenerate the performance of the subsequent SLU module. To
address this issue, word confusion networks (WCNs) have been used to encode the
input for SLU, which contain richer information than 1-best or n-best
hypotheses list. To further eliminate ambiguity, the last system act of
dialogue context is also utilized as additional input. In this paper, a novel
BERT based SLU model (WCN-BERT SLU) is proposed to encode WCNs and the dialogue
context jointly. It can integrate both structural information and ASR posterior
probabilities of WCNs in the BERT architecture. Experiments on DSTC2, a
benchmark of SLU, show that the proposed method is effective and can outperform
previous state-of-the-art models significantly.

本文提出了一种基于 BERT 的新型 SLU 模型（WCN-BERT SLU），其可以在 BERT 体系结构中整合 WCN 的结构信息和 ASR 后验概率，实验结果在 DSTC2 基准测试中表明该方法是有效的并且明显优于以前的最先进模型。

利用 BERT 联合编码词困惑网络和对话环境来进行口语理解

Jointly Encoding Word Confusion Network and Dialogue Context with BERT  for Spoken Language Understanding

This paper presents our novel method to encode word confusion networks, which
can represent a rich hypothesis space of automatic speech recognition systems,
via recurrent neural networks. We demonstrate the utility of our approach for
the task of dialog state tracking in spoken dialog systems that relies on
automatic speech recognition output. Encoding confusion networks outperforms
encoding the best hypothesis of the automatic speech recognition in a neural
system for dialog state tracking on the well-known second Dialog State Tracking
Challenge dataset.

本文介绍了一种用循环神经网络对单词混淆网络进行编码的新方法，该方法可以代表自动语音识别系统的丰富假设空间，在口语系统的对话状态跟踪任务中，编码混淆网络优于编码自动语音识别的最佳假设。