We introduce a new cross-modal fusion technique designed for generative error
correction in automatic speech recognition (ASR). Our methodology leverages
both acoustic information and external linguistic representations to generate
accurate speech transcription contexts. This marks a step towards a fresh
paradigm in generative error correction within the realm of n-best hypotheses.
Unlike the existing ranking-based rescoring methods, our approach adeptly uses
distinct initialization techniques and parameter-efficient algorithms to boost
ASR performance derived from pre-trained speech and text models. Through
evaluation across diverse ASR datasets, we evaluate the stability and
reproducibility of our fusion technique, demonstrating its improved word error
rate relative (WERR) performance in comparison to n-best hypotheses by
relatively 37.66%. To encourage future research, we have made our code and
pre-trained models open source at
this https URL

我们引入了一种新的跨模态融合技术，用于在自动语音识别（ASR）中进行生成性错误校正。我们的方法利用声学信息和外部语言表示来生成准确的语音转录上下文。与现有的基于排名的重排序方法不同，我们的方法巧妙地使用了不同的初始化技术和参数高效算法，从预训练的语音和文本模型中提高了 ASR 性能。通过在多样的 ASR 数据集上评估，我们评估了我们的融合技术的稳定性和可重现性，并相对于 n-best 假设的错误率（WERR）性能相对提高了 37.66%。为了鼓励未来的研究，我们将我们的代码和预训练模型开源。

耳语 LLaMA: 语音识别的跨模态生成错误修正框架

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework  for Speech Recognition

Spoken Language Understanding (SLU) systems parse speech into semantic
structures like dialog acts and slots. This involves the use of an Automatic
Speech Recognizer (ASR) to transcribe speech into multiple text alternatives
(hypotheses). Transcription errors, common in ASRs, impact downstream SLU
performance negatively. Approaches to mitigate such errors involve using richer
information from the ASR, either in form of N-best hypotheses or word-lattices.
We hypothesize that transformer models learn better with a simpler utterance
representation using the concatenation of the N-best ASR alternatives, where
each alternative is separated by a special delimiter [SEP]. In our work, we
test our hypothesis by using concatenated N-best ASR alternatives as the input
to transformer encoder models, namely BERT and XLM-RoBERTa, and achieve
performance equivalent to the prior state-of-the-art model on DSTC2 dataset. We
also show that our approach significantly outperforms the prior
state-of-the-art when subjected to the low data regime. Additionally, this
methodology is accessible to users of third-party ASR APIs which do not provide
word-lattice information.

使用自动语音识别提供的多个文本替代方案，采用简化的表述方式，通过 transformer 模型 BERT 和 XLM-RoBERTa 对 N-best 替代方案的连接作为输人，提高了 Spoken Language Understanding 系统的性能。