Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. However, when adapting speculative decoding to domain-specific target models, the acceptance rate of the generic draft model drops significantly due to domain shift. In this work, we systematically investigate knowledge distillation techniques for training domain draft models to improve their speculation accuracy. We compare white-box and black-box distillation approaches and explore their effectiveness in various data accessibility scenarios, including historical user queries, curated domain data, and synthetically generated alignment data. Our experiments across Function Calling, Biology, and Chinese domains show that offline distillation consistently outperforms online distillation by 11% to 25%, white-box distillation surpasses black-box distillation by 2% to 10%, and data scaling trends hold across domains. Additionally, we find that synthetic data can effectively align draft models and achieve 80% to 93% of the performance of training on historical user queries. These findings provide practical guidelines for training domain-specific draft models to improve speculative decoding efficiency.

本研究解决了在领域特定目标模型中使用推测解码时，通用草稿模型由于领域转变而接受率显著下降的问题。通过系统地研究知识蒸馏技术，提出了训练领域草稿模型以提高推测准确性的创新方法。实验结果表明，离线蒸馏超过在线蒸馏11%至25%，为训练领域特定草稿模型提供了实用指导，显著提高了推测解码的效率。

为推测解码训练领域草稿模型：最佳实践与见解