Large pre-trained language models (PLMs) have shown remarkable performance
across various natural language understanding (NLU) tasks, particularly in
low-resource settings. Nevertheless, their potential in Automatic Speech
Recognition (ASR) remains largely unexplored. This study investigates the
potential usage of PLMs for language modelling in ASR. We compare the
application of large-scale text sampling and probability conversion for
approximating GPT-2 into an n-gram model. Furthermore, we introduce a
vocabulary-restricted decoding method for random sampling, and evaluate the
effects of domain difficulty and data size on the usability of generated text.
Our findings across eight domain-specific corpora support the use of
sampling-based approximation and show that interpolating with a large sampled
corpus improves test perplexity over a baseline trigram by 15%. Our
vocabulary-restricted decoding method pushes this improvement further by 5% in
domain-specific settings.

本研究调查了预先训练的语言模型在自动语音识别中的潜在用途，对比了大规模文本抽样和概率转换的应用。在八个特定领域的语料库中，发现采样的近似方法支持使用，插值与大规模文本语料库一起使用对比基线三元组能使测试困惑度提高 15％，我们引入了一种有限制的词汇解码方法，这将进一步提高 5％的改进。