We tackle the challenging issue of aggressive fine-tuning encountered during
the process of transfer learning of pre-trained language models (PLMs) with
limited labeled downstream data. This problem primarily results in a decline in
performance on the subsequent task. Inspired by the adaptive boosting method in
traditional machine learning, we present an effective dynamic corrective
self-distillation (DCS) approach to improve the fine-tuning of the PLMs. Our
technique involves performing a self-distillation mechanism where, at each
iteration, the student model actively adapts and corrects itself by dynamically
adjusting the weights assigned to individual data points. This iterative
self-correcting process significantly enhances the overall fine-tuning
capability of PLMs, leading to improved performance and robustness. We
conducted comprehensive evaluations using the GLUE benchmark demonstrating the
efficacy of our method in enhancing the fine-tuning process for various PLMs
across diverse downstream tasks.

我们提出了一种有效的动态纠正自蒸馏方法来改善预训练语言模型的微调过程，通过迭代的自我纠正过程显著增强了预训练语言模型的微调能力，并在各种下游任务中提高了性能和鲁棒性。