Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.

Speculative Streaming是一种单一模型的推测解码方法，通过将起草融入目标模型，将微调目标从下一个标记预测改为未来n-gram预测的目标，以在总结、结构查询和语义表示等多个任务中提高1.8-3.1倍的解码速度，而不损失生成质量。此外，Speculative Streaming在参数效率方面表现出色，与Medusa-style架构相比，使用的额外参数较少，适用于资源受限的设备。

投机式流处理: 高速LLM推理无需辅助模型