We introduce ``Generative Fusion Decoding'' (GFD), a novel shallow fusion
framework, utilized to integrate Large Language Models (LLMs) into multi-modal
text recognition systems such as automatic speech recognition (ASR) and optical
character recognition (OCR). We derive the formulas necessary to enable GFD to
operate across mismatched token spaces of different models by mapping text
token space to byte token space, enabling seamless fusion during the decoding
process. The framework is plug-and-play, compatible with various
auto-regressive models, and does not require re-training for feature alignment,
thus overcoming limitations of previous fusion techniques. We highlight three
main advantages of GFD: First, by simplifying the complexity of aligning
different model sample spaces, GFD allows LLMs to correct errors in tandem with
the recognition model, reducing computation latencies. Second, the in-context
learning ability of LLMs is fully capitalized by GFD, increasing robustness in
long-form speech recognition and instruction aware speech recognition. Third,
GFD enables fusing recognition models deficient in Chinese text recognition
with LLMs extensively trained on Chinese. Our evaluation demonstrates that GFD
significantly improves performance in ASR and OCR tasks, with ASR reaching
state-of-the-art in the NTUML2021 benchmark. GFD provides a significant step
forward in model integration, offering a unified solution that could be widely
applicable to leveraging existing pre-trained models through step by step
fusion.

我们引入了 “Generative Fusion Decoding”（GFD），这是一种新颖的浅层融合框架，用于将大型语言模型（LLMs）集成到自动语音识别（ASR）和光学字符识别（OCR）等多模态文本识别系统中。通过将文本令牌空间映射到字节令牌空间，我们推导出了启用 GFD 在不同模型的不匹配令牌空间上运行所需的公式，从而在解码过程中实现无缝融合。该框架是即插即用的，与各种自回归模型兼容，不需要重新训练以进行特征对齐，从而克服了以前融合技术的局限性。我们强调了 GFD 的三个主要优势：首先，通过简化对齐不同模型样本空间的复杂性，GFD 使 LLMs 能够与识别模型一起纠正错误，减少计算延迟。其次，GFD 充分利用了 LLMs 的上下文学习能力，增加了在长篇演讲识别和指令感知语音识别中的鲁棒性。第三，GFD 使得与 LLMs 广泛训练的、在中文识别方面存在不足的识别模型进行融合成为可能。我们的评估表明，GFD 在 ASR 和 OCR 任务中显著提高了性能，其中 ASR 在 NTUML2021 基准中达到了最先进水平。GFD 在模型集成方面迈出了重要的一步，提供了一种统一的解决方案，可以通过逐步融合来广泛适用于利用现有预训练模型。