Autoregressive decoding of large language models (LLMs) is memory bandwidth
bounded, resulting in high latency and significant wastes of the parallel
processing power of modern accelerators. Existing methods for accelerating LLM
decoding often require a draft model (e.g., speculative decoding), which is
nontrivial to obtain and unable to generalize. In this paper, we introduce
Lookahead decoding, an exact, parallel decoding algorithm that accelerates LLM
decoding without needing auxiliary models or data stores. It allows trading
per-step log(FLOPs) to reduce the number of total decoding steps, is more
parallelizable on single or multiple modern accelerators, and is compatible
with concurrent memory-efficient attention (e.g., FlashAttention). Our
implementation of Lookahead decoding can speed up autoregressive decoding by up
to 1.8x on MT-bench and 4x with strong scaling on multiple GPUs in code
completion tasks. Our code is avialable at
this https URL

使用准确并行的预读解码算法加速大型语言模型的自回归解码，从而减少总解码步骤，提高解码速度并充分发挥现代加速器的并行处理能力。