The recent advancements in large language models (LLMs) have been
extraordinary, yet the escalating inference costs associated with them present
challenges in real-world applications. To address these challenges, we propose
a novel approach called Early-exiting Speculative Decoding (EESD) with lossless
acceleration. Specifically, EESD utilizes a segment of the LLM to generate
draft tokens, incorporating Early-exiting structures after the first N layers.
To enhance the quality of draft tokens, a self-distillation method is
integrated. This early-exiting design not only reduces deployment and training
costs but also significantly accelerates the token generation speed. Moreover,
we introduce a novel sampling mechanism that leverages Thompson Sampling to
regulate the generation processes, automatically determining the quantity of
draft tokens in each round. The original LLM is then employed to validate these
draft tokens through a single forward pass, and thus guarantees that the final
output text maintains a distribution consistent with vanilla auto-regressive
decoding. The experimental results on both 13B and 70B models demonstrate that
our approach decodes tokens at a markedly accelerated rate compared to prior
methods, showing the effectiveness of our approach.

在大型语言模型中，我们提出了一种新的方法，即具有无损加速的早期退出推理（EESD），通过在前 N 层后引入早期退出结构，利用语言模型的一部分生成初步令牌，并通过自蒸馏方法提高初步令牌的质量。我们还引入了一种新的采样机制，利用汤普森采样调节生成过程，自动确定每一轮的初步令牌数量。实验结果表明，与先前的方法相比，我们的方法在解码令牌时具有明显的加速效果。

通过提前退出进行投机解码以加速 LLM 推断，并采用汤普森抽样控制机制

Speculative Decoding via Early-exiting for Faster LLM Inference with  Thompson Sampling Control Mechanism

Recently, transformer-based methods have achieved state-of-the-art prediction
quality on human pose estimation(HPE). Nonetheless, most of these
top-performing transformer-based models are too computation-consuming and
storage-demanding to deploy on edge computing platforms. Those
transformer-based models that require fewer resources are prone to
under-fitting due to their smaller scale and thus perform notably worse than
their larger counterparts. Given this conundrum, we introduce SDPose, a new
self-distillation method for improving the performance of small
transformer-based models. To mitigate the problem of under-fitting, we design a
transformer module named Multi-Cycled Transformer(MCT) based on multiple-cycled
forwards to more fully exploit the potential of small model parameters.
Further, in order to prevent the additional inference compute-consuming brought
by MCT, we introduce a self-distillation scheme, extracting the knowledge from
the MCT module to a naive forward model. Specifically, on the MSCOCO validation
dataset, SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs.
Furthermore, SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset
with 6.2M parameters and 4.7 GFLOPs, achieving a new state-of-the-art among
predominant tiny neural network methods. Our code is available at
this https URL

SDPose 是一种新的自我蒸馏方法，它基于多周期前向传播设计了一个名为 MCT 的 transformer 模块来提高小型 transformer 模型的性能，同时通过将 MCT 模块的知识提取到一个简单的前向模型中，避免了额外的计算。