Are n-gram language models still relevant in this era of neural large
language models (LLMs)? Our answer is yes, and we show their values in both
text analysis and improving neural LLMs. Yet this necessitates modernizing
n-gram models in two aspects. First, we train them at the same data scale as
neural LLMs -- 1.4 trillion tokens. This is the largest n-gram model ever
built. Second, existing n-gram models use small n which hinders their
performance; we instead allow n to be arbitrarily large, by introducing a new
$\infty$-gram LM with backoff. Instead of pre-computing n-gram count tables
(which would be very expensive), we develop an engine named infini-gram --
powered by suffix arrays -- that can compute $\infty$-gram (as well as n-gram
with arbitrary n) probabilities with millisecond-level latency. The
$\infty$-gram framework and infini-gram engine enable us to conduct many novel
and interesting analyses of human-written and machine-generated text: we find
that the $\infty$-gram LM has fairly high accuracy for next-token prediction
(47%), and can complement neural LLMs to greatly reduce their language modeling
perplexities. When analyzing machine-generated text, we also observe
irregularities in the machine--$\infty$-gram agreement level with respect to
the suffix length, which indicates deficiencies in neural LLM pretraining and
the positional embeddings of Transformers. We open-source our infini-gram
engine in the hopes of enabling more study on how to best use verbatim
information retrieved from large text corpora.

n-gram 语言模型在神经网络大型语言模型时代仍然具有重要意义，本研究展示了其在文本分析和改进神经网络大型语言模型中的价值，并从两个方面对 n-gram 模型进行现代化改进：在与神经网络大型语言模型相同的数据规模下进行训练，建立最大的 n-gram 模型，以及允许 n 的大小任意增加，并引入具备毫秒级延迟的 suffix 数组驱动引擎 infini-gram 来计算无限阶的 n-gram 概率。