In recent years, knowledge graph completion (KGC) models based on pre-trained
language model (PLM) have shown promising results. However, the large number of
parameters and high computational cost of PLM models pose challenges for their
application in downstream tasks. This paper proposes a progressive distillation
method based on masked generation features for KGC task, aiming to
significantly reduce the complexity of pre-trained models. Specifically, we
perform pre-distillation on PLM to obtain high-quality teacher models, and
compress the PLM network to obtain multi-grade student models. However,
traditional feature distillation suffers from the limitation of having a single
representation of information in teacher models. To solve this problem, we
propose masked generation of teacher-student features, which contain richer
representation information. Furthermore, there is a significant gap in
representation ability between teacher and student. Therefore, we design a
progressive distillation method to distill student models at each grade level,
enabling efficient knowledge transfer from teachers to students. The
experimental results demonstrate that the model in the pre-distillation stage
surpasses the existing state-of-the-art methods. Furthermore, in the
progressive distillation stage, the model significantly reduces the model
parameters while maintaining a certain level of performance. Specifically, the
model parameters of the lower-grade student model are reduced by 56.7\%
compared to the baseline.

本文提出了一种基于掩码生成特征的渐进蒸馏方法，用于知识图谱完成任务，显著降低了预训练模型的复杂性。通过对预训练模型进行预蒸馏以获取高质量的教师模型，然后压缩预训练模型网络获取多级学生模型，通过渐进蒸馏实现了从教师到学生的高效知识转移。实验结果表明，在预蒸馏阶段，模型超过了现有最先进方法，在渐进蒸馏阶段，模型在显著减少模型参数的同时保持一定水平的性能。具体而言，相较于基线，低级别学生模型的模型参数减少了 56.7%。