The high computational and memory requirements of generative large language
models (LLMs) make it challenging to serve them quickly and cheaply. This paper
introduces SpecInfer, an LLM serving system that accelerates generative LLM
inference with speculative inference and token tree verification. A key insight
behind SpecInfer is to combine various collectively boost-tuned small language
models to jointly predict the LLM's outputs; the predictions are organized as a
token tree, whose nodes each represent a candidate token sequence. The
correctness of all candidate token sequences represented by a token tree is
verified by the LLM in parallel using a novel tree-based parallel decoding
mechanism. SpecInfer uses an LLM as a token tree verifier instead of an
incremental decoder, which significantly reduces the end-to-end latency and
computational requirement for serving generative LLMs while provably preserving
model quality.

本文介绍了 SpecInfer，一种使用预测推断和令牌树验证加速生成式大型语言模型推断的 LLM 服务系统，其使用多个小型语言模型共同预测 LLM 的输出，并使用一种新的基于树的并行解码机制验证所有候选的令牌序列的正确性。通过使用 LLM 作为令牌树验证器，而不是增量解码器，SpecInfer 显着降低了用于服务 LLM 的端到端延迟和计算要求，同时确保保持模型质量。