We propose an acceleration scheme for large language models (LLMs) through
Speculative Decoding with Semantic Adaptive Tokens (SDSAT). The primary
objective of this design is to enhance the LLM model's ability to generate
draft tokens more accurately without compromising the model's accuracy. The
core strategies involve: 1) Fine-tune the model by incorporating semantic
adaptive tokens that possess flexible decoding capabilities without changing
its structure, allowing them to generate high-quality draft tokens. 2) By
employing a training method that does not affect the standard tokens, the model
can acquire parallel decoding abilities atop its original framework with
minimal training overhead. 3) We have designed the "two-step-draft-then-verify"
generation strategies using both greedy search and nucleus sampling.
Experiments conducted on the CodeLlama-13B and 7B models have yielded speed
increases of over 3.5X and 3.0X, respectively. Please refer to
this https URL

通过引入具有语义自适应能力的令牌，提出一种用于大型语言模型（LLM）的加速方案（SDSAT）。该方案通过细调模型和使用训练方法，以及使用新的生成策略，能够在不影响模型准确性的情况下，提高模型生成草稿令牌的准确性，并且在 CodeLlama-13B 和 7B 模型上实验，分别获得了超过 3.5X 和 3.0X 的加速效果。