In the era of large models, the autoregressive nature of decoding often
results in latency serving as a significant bottleneck. We propose a
non-autoregressive LM-fused ASR system that effectively leverages the
parallelization capabilities of accelerator hardware. Our approach combines the
Universal Speech Model (USM) and the PaLM 2 language model in per-segment
scoring mode, achieving an average relative WER improvement across all
languages of 10.8% on FLEURS and 3.6% on YouTube captioning. Furthermore, our
comprehensive ablation study analyzes key parameters such as LLM size, context
length, vocabulary size, fusion methodology. For instance, we explore the
impact of LLM size ranging from 128M to 340B parameters on ASR performance.
This study provides valuable insights into the factors influencing the
effectiveness of practical large-scale LM-fused speech recognition systems.

在大模型时代，解码的自回归特性通常导致延迟成为一个重要瓶颈。我们提出了一个非自回归 LM 融合 ASR 系统，有效地利用了加速器硬件的并行化能力。我们的方法是将 USM 和 PaLM 2 语言模型以每个片段评分模式结合起来，在 FLEURS 和 YouTube 字幕上实现了相对 WER 的平均改进 10.8％和 3.6％。此外，我们的综合消融研究分析了关键参数，如 LLM 大小、上下文长度、词汇大小、融合方法等。例如，我们探讨了 LLM 大小从 128M 到 340B 参数对 ASR 性能的影响。该研究为实际大规模 LM 融合语音识别系统的有效性提供了有价值的见解。