One of the most striking findings in modern research on large language models
(LLMs) is that scaling up compute during training leads to better results.
However, less attention has been given to the benefits of scaling compute
during inference. This survey focuses on these inference-time approaches. We
explore three areas under a unified mathematical formalism: token-level
generation algorithms, meta-generation algorithms, and efficient generation.
Token-level generation algorithms, often called decoding algorithms, operate by
sampling a single token at a time or constructing a token-level search space
and then selecting an output. These methods typically assume access to a
language model's logits, next-token distributions, or probability scores.
Meta-generation algorithms work on partial or full sequences, incorporating
domain knowledge, enabling backtracking, and integrating external information.
Efficient generation methods aim to reduce token costs and improve the speed of
generation. Our survey unifies perspectives from three research communities:
traditional natural language processing, modern LLMs, and machine learning
systems.

通过对大型语言模型的研究，发现在训练过程中提高计算能力可以取得更好的结果，然而对于推断阶段提高计算能力的好处却没有得到足够的关注。本文调查了推断阶段的几种方法，包括基于令牌级别的生成算法、元生成算法和高效生成方法，并从传统自然语言处理、现代大型语言模型和机器学习系统的角度统一了观点。