Speculative decoding (SD) has attracted a significant amount of research
attention due to the substantial speedup it can achieve for LLM inference.
However, despite the high speedups they offer, speculative decoding methods
often achieve optimal performance on high-end devices or with a substantial GPU
memory overhead. Given limited memory and the necessity of quantization, a
high-performing model on a high-end GPU can slow down by up to 7 times. To this
end, we propose Skippy Simultaneous Speculative Decoding (or S3D), a
cost-effective self-speculative SD method based on simultaneous multi-token
decoding and mid-layer skipping. When compared against recent effective
open-source SD systems, our method has achieved one of the top
performance-memory ratios while requiring minimal architecture changes and
training data. Leveraging our memory efficiency, we created a smaller yet more
effective SD model based on Phi-3. It is 1.4 to 2 times faster than the
quantized EAGLE model and operates in half-precision while using less VRAM.

Skippy Simultaneous Speculative Decoding (S3D) 是一种成本有效、基于同时多令牌解码和中间层跳跃的自我推测 SD 方法，与最近的有效开源 SD 系统相比，它在性能 - 内存比上取得了最佳表现，而且只需要最小的架构改变和训练数据。利用我们的内存效率，我们基于 Phi-3 创建了一个更小、更高效的 SD 模型，它比量化的 EAGLE 模型快 1.4 到 2 倍，使用半精度操作，同时使用更少的 VRAM。