Recent recommender systems have shown remarkable performance by using an
ensemble of heterogeneous models. However, it is exceedingly costly because it
requires resources and inference latency proportional to the number of models,
which remains the bottleneck for production. Our work aims to transfer the
ensemble knowledge of heterogeneous teachers to a lightweight student model
using knowledge distillation (KD), to reduce the huge inference costs while
retaining high accuracy. Through an empirical study, we find that the efficacy
of distillation severely drops when transferring knowledge from heterogeneous
teachers. Nevertheless, we show that an important signal to ease the difficulty
can be obtained from the teacher's training trajectory. This paper proposes a
new KD framework, named HetComp, that guides the student model by transferring
easy-to-hard sequences of knowledge generated from the teachers' trajectories.
To provide guidance according to the student's learning state, HetComp uses
dynamic knowledge construction to provide progressively difficult ranking
knowledge and adaptive knowledge transfer to gradually transfer finer-grained
ranking information. Our comprehensive experiments show that HetComp
significantly improves the distillation quality and the generalization of the
student model.

本研究提出了一种名为 HetComp 的知识蒸馏框架，用于将异构模型的整体知识转移给一个轻量级模型，以减少资源成本和推理延迟，该框架使用动态知识构建和自适应知识传输提供逐步更难的排序信息，并通过全面实验表明该框架能够显著提高精度和泛化性能。