Existing speculative decoding methods typically require additional model
structure and training processes to assist the model for draft token
generation. This makes the migration of acceleration methods to the new model
more costly and more demanding on device memory. To address this problem, we
propose the Make Some Noise (MSN) training framework as a replacement for the
supervised fine-tuning stage of the large language model. The training method
simply introduces some noise at the input for the model to learn the denoising
task. It significantly enhances the parallel decoding capability of the model
without affecting the original task capability. In addition, we propose a
tree-based retrieval-augmented Jacobi (TR-Jacobi) decoding strategy to further
improve the inference speed of MSN models. Experiments in both the general and
code domains have shown that MSN can improve inference speed by 2.3-2.7x times
without compromising model performance. The MSN model also achieves comparable
acceleration ratios to the SOTA model with additional model structure on
Spec-Bench.

通过引入噪声训练模型的 Make Some Noise（MSN）框架和基于树的检索增强 Jacobi（TR-Jacobi）解码策略，可以改进大型语言模型的推断速度，提高并行解码能力，并且通过在不影响模型性能的情况下在推断阶段获得 2.3-2.7 倍的加速比。

噪声训练解锁语言模型并行推理能力

Make Some Noise: Unlocking Language Model Parallel Inference Capability  through Noisy Training

Recent research (arXiv:2310.11453, arXiv:2402.17764) has proposed binary and
ternary transformer networks as a way to significantly reduce memory and
improve inference speed in Large Language Models (LLMs) while maintaining
accuracy. In this work, we apply techniques from mechanistic interpretability
to investigate whether such networks learn distinctly different or similar
algorithms when compared to full-precision transformer networks. In particular,
we reverse engineer the algorithms learned for the toy problem of modular
addition where we find that binary and ternary networks learn similar
algorithms as full precision networks. This provides evidence against the
possibility of using binary and ternary networks as a more interpretable
alternative in the LLM setting.

研究表明，将二进制和三进制变压器网络应用于大型语言模型能够显著减少内存并提高推理速度，并且研究结果显示，这些网络在学习模块化加法时与全精度变压器网络学习的算法相似，因此无法作为语言模型中更可解释的替代方案。