Large language models (LLMs) have revolutionized natural language processing
and broadened their applicability across diverse commercial applications.
However, the deployment of these models is constrained by high inference time
in multilingual settings. To mitigate this challenge, this paper explores a
training recipe of an assistant model in speculative decoding, which are
leveraged to draft and-then its future tokens are verified by the target LLM.
We show that language-specific draft models, optimized through a targeted
pretrain-and-finetune strategy, substantially brings a speedup of inference
time compared to the previous methods. We validate these models across various
languages in inference time, out-of-domain speedup, and GPT-4o evaluation.

大语言模型在自然语言处理中产生了革命性的作用，并且扩展了它在不同商业应用中的适用性。然而，这些模型在多语言环境中的部署受到推理时间的限制。为了缓解这一挑战，本文探讨了使用推测解码的助理模型的训练方法，其中助理模型用于草拟未来的令牌，并通过目标语言模型进行验证。我们表明，通过有针对性的预训练和微调策略优化的专门针对语言的草拟模型，在推理时间上显著减少了时间，相较于以前的方法。我们在推理时间、领域外优化以及 GPT-4o 评估等方面验证了这些模型。

迈向快速多语言 LLM 推断：投机式解码和专业的起草机

Towards Fast Multilingual LLM Inference: Speculative Decoding and  Specialized Drafters

Speculative decoding is a relatively new decoding framework that leverages
small and efficient draft models to reduce the latency of LLMs. In this study,
we introduce GliDe and CaPE, two low-hassle modifications to vanilla
speculative decoding to further improve the decoding speed of a frozen LLM.
Specifically, GliDe is a modified draft model architecture that reuses the
cached keys and values from the target LLM, while CaPE is a proposal expansion
method that uses the draft model's confidence scores to help select additional
candidate tokens for verification. Extensive experiments on different
benchmarks demonstrate that our proposed GliDe draft model significantly
reduces the expected decoding latency. Additional evaluation using walltime
reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend
the improvement to 2.61x with CaPE. We will release our code, data, and the
trained draft models.

该研究介绍了两种改进的解码模型，GliDe 和 CaPE，以进一步提高冻结 LLM 的解码速度，并通过大量实验证明 GliDe 可以显著降低期望的解码延迟。