We introduce a new cross-modal fusion technique designed for generative error
correction in automatic speech recognition (ASR). Our methodology leverages
both acoustic information and external linguistic representations to generate
accurate speech transcription contexts. This marks a step towards a fresh
paradigm in generative error correction within the realm of n-best hypotheses.
Unlike the existing ranking-based rescoring methods, our approach adeptly uses
distinct initialization techniques and parameter-efficient algorithms to boost
ASR performance derived from pre-trained speech and text models. Through
evaluation across diverse ASR datasets, we evaluate the stability and
reproducibility of our fusion technique, demonstrating its improved word error
rate relative (WERR) performance in comparison to n-best hypotheses by
relatively 37.66%. To encourage future research, we have made our code and
pre-trained models open source at
this https URL

我们引入了一种新的跨模态融合技术，用于在自动语音识别（ASR）中进行生成性错误校正。我们的方法利用声学信息和外部语言表示来生成准确的语音转录上下文。与现有的基于排名的重排序方法不同，我们的方法巧妙地使用了不同的初始化技术和参数高效算法，从预训练的语音和文本模型中提高了 ASR 性能。通过在多样的 ASR 数据集上评估，我们评估了我们的融合技术的稳定性和可重现性，并相对于 n-best 假设的错误率（WERR）性能相对提高了 37.66%。为了鼓励未来的研究，我们将我们的代码和预训练模型开源。