Knowledge distillation typically transfers knowledge from a teacher model to a student model by minimizing differences between their output distributions. However, existing distillation approaches largely focus on mimicking absolute probabilities and neglect the valuable relational inductive biases embedded in the teacher's relative predictions, leading to exposure bias. In this paper, we propose Group Relative Knowledge Distillation (GRKD), a novel framework that distills teacher knowledge by learning the relative ranking among classes, rather than directly fitting the absolute distribution. Specifically, we introduce a group relative loss that encourages the student model to preserve the pairwise preference orderings provided by the teacher's outputs. Extensive experiments on classification benchmarks demonstrate that GRKD achieves superior generalization compared to existing methods, especially in tasks requiring fine-grained class differentiation. Our method provides a new perspective on exploiting teacher knowledge, focusing on relational structure rather than absolute likelihood.

本研究针对现有知识蒸馏方法忽视教师模型相对预测中重要的关系性归纳偏置的问题，提出了群体相对知识蒸馏（GRKD）框架。GRKD通过关注类别之间的相对排名来提炼教师知识，实验表明该方法在需要细粒度分类的任务中相较于现有方法表现出更优的泛化能力，展示了利用教师知识的新视角。

群体相对知识蒸馏：学习教师的关系性归纳偏置