Large language models have shown impressive capabilities across a variety of
NLP tasks, yet their generating text autoregressively is time-consuming. One
way to speed them up is speculative decoding, which generates candidate
segments (a sequence of tokens) from a fast draft model that is then verified
in parallel by the target model. However, the acceptance rate of candidate
tokens receives limitations from several factors, such as the model, the
dataset, and the decoding setup. This paper proposes sampling multiple
candidates from a draft model and then organising them in batches for
verification. We design algorithms for efficient multi-candidate verification
while maintaining the distribution of the target model. Our approach shows
significant improvements in acceptance rates on multiple datasets and models,
consistently outperforming standard speculative decoding.

大型语言模型在各种自然语言处理任务中展现出令人印象深刻的能力，但是自回归生成文本却非常耗时。提高速度的一个方法是进行猜测性解码，即由快速的草稿模型生成候选分段（一系列令牌），然后由目标模型并行验证。然而，候选标记的接受率受到模型、数据集和解码设置等多个因素的限制。本文提出了从草稿模型中采样多个候选分段，并将它们分批进行验证的方法。我们设计了高效的多候选验证算法，同时保持目标模型的分布。我们的方法在多个数据集和模型上都展现出显著的接受率改进，始终优于标准的猜测性解码。