Despite the empirical success and practical significance of (relational)
knowledge distillation that matches (the relations of) features between teacher
and student models, the corresponding theoretical interpretations remain
limited for various knowledge distillation paradigms. In this work, we take an
initial step toward a theoretical understanding of relational knowledge
distillation (RKD), with a focus on semi-supervised classification problems. We
start by casting RKD as spectral clustering on a population-induced graph
unveiled by a teacher model. Via a notion of clustering error that quantifies
the discrepancy between the predicted and ground truth clusterings, we
illustrate that RKD over the population provably leads to low clustering error.
Moreover, we provide a sample complexity bound for RKD with limited unlabeled
samples. For semi-supervised learning, we further demonstrate the label
efficiency of RKD through a general framework of cluster-aware semi-supervised
learning that assumes low clustering errors. Finally, by unifying data
augmentation consistency regularization into this cluster-aware framework, we
show that despite the common effect of learning accurate clusterings, RKD
facilitates a "global" perspective through spectral clustering, whereas
consistency regularization focuses on a "local" perspective via expansion.

关于关系性知识蒸馏的理论解释和在半监督分类问题中的应用，本文通过将关系性知识蒸馏转化为基于教师模型的谱聚类，并量化聚类误差来证明其有效性，并通过示例复杂度界限和群集感知的半监督学习框架进一步展示了其高标记效率。此外，通过将数据增强一致性正则化与群集感知框架统一，发现关系性知识蒸馏通过谱聚类提供了 “全局” 视角，而一致性正则化则聚焦于 “局部” 视角的扩展。

集群感知半监督学习：可证明学习聚类的关系知识蒸馏

Cluster-aware Semi-supervised Learning: Relational Knowledge  Distillation Provably Learns Clustering

Class-Incremental Learning (CIL) struggles with catastrophic forgetting when
learning new knowledge, and Data-Free CIL (DFCIL) is even more challenging
without access to the training data of previously learned classes. Though
recent DFCIL works introduce techniques such as model inversion to synthesize
data for previous classes, they fail to overcome forgetting due to the severe
domain gap between the synthetic and real data. To address this issue, this
paper proposes relation-guided representation learning (RRL) for DFCIL, dubbed
R-DFCIL. In RRL, we introduce relational knowledge distillation to flexibly
transfer the structural relation of new data from the old model to the current
model. Our RRL-boosted DFCIL can guide the current model to learn
representations of new classes better compatible with representations of
previous classes, which greatly reduces forgetting while improving plasticity.
To avoid the mutual interference between representation and classifier
learning, we employ local rather than global classification loss during RRL.
After RRL, the classification head is refined with global class-balanced
classification loss to address the data imbalance issue as well as learn the
decision boundaries between new and previous classes. Extensive experiments on
CIFAR100, Tiny-ImageNet200, and ImageNet100 demonstrate that our R-DFCIL
significantly surpasses previous approaches and achieves a new state-of-the-art
performance for DFCIL. Code is available at
this https URL

本文提出了一种基于关系指导的表征学习方法，用于解决数据自由的类增量学习（DFCIL）中出现的遗忘问题，并展示了其在三个数据集上实现了新的最优性能。

R-DFCIL: 基于关系引导的数据无关类增量学习表征学习

R-DFCIL: Relation-Guided Representation Learning for Data-Free Class  Incremental Learning

Entity alignment aims at integrating heterogeneous knowledge from different
knowledge graphs. Recent studies employ embedding-based methods by first
learning the representation of Knowledge Graphs and then performing entity
alignment via measuring the similarity between entity embeddings. However, they
failed to make good use of the relation semantic information due to the
trade-off problem caused by the different objectives of learning knowledge
embedding and neighborhood consensus. To address this problem, we propose
Relational Knowledge Distillation for Entity Alignment (RKDEA), a Graph
Convolutional Network (GCN) based model equipped with knowledge distillation
for entity alignment. We adopt GCN-based models to learn the representation of
entities by considering the graph structure and incorporating the relation
semantic information into GCN via knowledge distillation. Then, we introduce a
novel adaptive mechanism to transfer relational knowledge so as to jointly
learn entity embedding and neighborhood consensus. Experimental results on
several benchmarking datasets demonstrate the effectiveness of our proposed
model.

本文提出了一种基于图卷积神经网络和知识蒸馏的实体对齐模型以解决关系语义信息利用不充分的问题，并通过实验验证了其有效性。

关系知识蒸馏下联合学习知识嵌入和邻域一致性以实现实体对齐

Jointly Learning Knowledge Embedding and Neighborhood Consensus with Relational Knowledge Distillation for Entity Alignment

Knowledge distillation aims at transferring knowledge acquired in one model
(a teacher) to another model (a student) that is typically smaller. Previous
approaches can be expressed as a form of training the student to mimic output
activations of individual data examples represented by the teacher. We
introduce a novel approach, dubbed relational knowledge distillation (RKD),
that transfers mutual relations of data examples instead. For concrete
realizations of RKD, we propose distance-wise and angle-wise distillation
losses that penalize structural differences in relations. Experiments conducted
on different tasks show that the proposed method improves educated student
models with a significant margin. In particular for metric learning, it allows
students to outperform their teachers' performance, achieving the state of the
arts on standard benchmark datasets.

该论文提出了关系知识蒸馏方法（RKD），用于将数据示例之间的相互关系转移给学生模型，进而提高其在度量学习等任务中的性能，尤其是在标准基准数据集上取得了超越其老师的表现。