Efficient methods for storing and querying are critical for scaling
high-order n-gram language models to large corpora. We propose a language model
based on compressed suffix trees, a representation that is highly compact and
can be easily held in memory, while supporting queries needed in computing
language model probabilities on-the-fly. We present several optimisations which
improve query runtimes up to 2500x, despite only incurring a modest increase in
construction time and memory usage. For large corpora and high Markov orders,
our method is highly competitive with the state-of-the-art KenLM package. It
imposes much lower memory requirements, often by orders of magnitude, and has
runtimes that are either similar (for training) or comparable (for querying).

本文介绍一种基于压缩后缀树的语言模型，该模型具有高度紧凑的表现形式，可在内存中轻松容纳，并支持在计算语言模型概率时所需的查询。该模型在构建时间和内存使用方面仅会导致适度的增加，但是可以通过多种优化方式提高查询运行时间，达到高达 2500 倍的提速效果。在大规模语料库和高马尔可夫阶数的应用场景下，该方法与现有的 KenLM 算法相比具有高度的竞争力，并且所需的内存要求通常低好几个数量级，查询运行时间相当或略快。