Recent advancements in generative large language models (LLMs) have
significantly boosted the performance in natural language processing tasks.
However, their efficiency is hampered by the inherent limitations in
autoregressive token generation. While parallel decoding with token tree
verification, e.g., Medusa, has been proposed to improve decoding parallelism
and efficiency, it often struggles with maintaining contextual relationships
due to its independent token prediction approach and incurs significant
verification overhead, especially with large tree sizes and batch processing.
In this paper, we propose ProPD, an efficient LLM parallel decoding framework
based on dynamic token tree pruning and generation. ProPD features an advanced
early pruning mechanism to efficiently eliminate unpromising token sequences to
improve verification efficiency. Additionally, it introduces a dynamic token
tree generation algorithm to balance the computation and parallelism of the
verification phase in real-time and maximize the overall efficiency across
different batch sizes, sequence lengths, and tasks, etc. We verify ProPD across
a diverse set of datasets, LLMs, and batch sizes and demonstrate ProPD
consistently outperforms existing decoding algorithms by 1.1-3.2x.

提出了一种基于动态标记树修剪和生成的高效的大型语言模型（LLM）并行解码框架 ProPD，它通过提高并行性和效率来改进解码的性能和上下文关系维护问题，并在各种批量大小、序列长度和任务等方面最大化整体效率。在各种数据集、LLM 和批量大小上验证 ProPD，并证明其性能始终优于现有的解码算法 1.1-3.2 倍。

ProPD: 动态令牌树剪枝和生成用于 LLM 并行解码

ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel  Decoding

The high computational and memory requirements of generative large language
models (LLMs) make it challenging to serve them quickly and cheaply. This paper
introduces SpecInfer, an LLM serving system that accelerates generative LLM
inference with speculative inference and token tree verification. A key insight
behind SpecInfer is to combine various collectively boost-tuned small language
models to jointly predict the LLM's outputs; the predictions are organized as a
token tree, whose nodes each represent a candidate token sequence. The
correctness of all candidate token sequences represented by a token tree is
verified by the LLM in parallel using a novel tree-based parallel decoding
mechanism. SpecInfer uses an LLM as a token tree verifier instead of an
incremental decoder, which significantly reduces the end-to-end latency and
computational requirement for serving generative LLMs while provably preserving
model quality.

本文介绍了 SpecInfer，一种使用预测推断和令牌树验证加速生成式大型语言模型推断的 LLM 服务系统，其使用多个小型语言模型共同预测 LLM 的输出，并使用一种新的基于树的并行解码机制验证所有候选的令牌序列的正确性。通过使用 LLM 作为令牌树验证器，而不是增量解码器，SpecInfer 显着降低了用于服务 LLM 的端到端延迟和计算要求，同时确保保持模型质量。