Recently, transformer-based methods have achieved state-of-the-art prediction
quality on human pose estimation(HPE). Nonetheless, most of these
top-performing transformer-based models are too computation-consuming and
storage-demanding to deploy on edge computing platforms. Those
transformer-based models that require fewer resources are prone to
under-fitting due to their smaller scale and thus perform notably worse than
their larger counterparts. Given this conundrum, we introduce SDPose, a new
self-distillation method for improving the performance of small
transformer-based models. To mitigate the problem of under-fitting, we design a
transformer module named Multi-Cycled Transformer(MCT) based on multiple-cycled
forwards to more fully exploit the potential of small model parameters.
Further, in order to prevent the additional inference compute-consuming brought
by MCT, we introduce a self-distillation scheme, extracting the knowledge from
the MCT module to a naive forward model. Specifically, on the MSCOCO validation
dataset, SDPose-T obtains 69.7% mAP with 4.4M parameters and 1.8 GFLOPs.
Furthermore, SDPose-S-V2 obtains 73.5% mAP on the MSCOCO validation dataset
with 6.2M parameters and 4.7 GFLOPs, achieving a new state-of-the-art among
predominant tiny neural network methods. Our code is available at
this https URL

SDPose 是一种新的自我蒸馏方法，它基于多周期前向传播设计了一个名为 MCT 的 transformer 模块来提高小型 transformer 模型的性能，同时通过将 MCT 模块的知识提取到一个简单的前向模型中，避免了额外的计算。