Knowledge amalgamation (KA) aims to learn a compact student model to handle
the joint objective from multiple teacher models that are are specialized for
their own tasks respectively. Current methods focus on coarsely aligning
teachers and students in the common representation space, making it difficult
for the student to learn the proper decision boundaries from a set of
heterogeneous teachers. Besides, the KL divergence in previous works only
minimizes the probability distribution difference between teachers and the
student, ignoring the intrinsic characteristics of teachers. Therefore, we
propose a novel Contrastive Knowledge Amalgamation (CKA) framework, which
introduces contrastive losses and an alignment loss to achieve intra-class
cohesion and inter-class separation.Contrastive losses intra- and inter- models
are designed to widen the distance between representations of different
classes. The alignment loss is introduced to minimize the sample-level
distribution differences of teacher-student models in the common representation
space.Furthermore, the student learns heterogeneous unsupervised classification
tasks through soft targets efficiently and flexibly in the task-level
amalgamation. Extensive experiments on benchmarks demonstrate the
generalization capability of CKA in the amalgamation of specific task as well
as multiple tasks. Comprehensive ablation studies provide a further insight
into our CKA.

知识融合旨在学习一个紧凑的学生模型，以处理多个针对各自任务的教师模型的联合目标。我们提出了一种新颖的对比知识融合（CKA）框架，通过引入对比损失和对齐损失来实现类内凝聚和类间分离，以使学生能够学习多个异构教师自任务的适当决策边界。