Knowledge Distillation (KD) aims to transfer knowledge in a teacher-student
framework, by providing the predictions of the teacher network to the student
network in the training stage to help the student network generalize better. It
can use either a teacher with high capacity or {an} ensemble of multiple
teachers. However, the latter is not convenient when one wants to use
feature-map-based distillation methods. For a solution, this paper proposes a
versatile and powerful training algorithm named FEature-level Ensemble for
knowledge Distillation (FEED), which aims to transfer the ensemble knowledge
using multiple teacher networks. We introduce a couple of training algorithms
that transfer ensemble knowledge to the student at the feature map level. Among
the feature-map-based distillation methods, using several non-linear
transformations in parallel for transferring the knowledge of the multiple
teacher{s} helps the student find more generalized solutions. We name this
method as parallel FEED, andexperimental results on CIFAR-100 and ImageNet show
that our method has clear performance enhancements, without introducing any
additional parameters or computations at test time. We also show the
experimental results of sequentially feeding teacher's information to the
student, hence the name sequential FEED, and discuss the lessons obtained.
Additionally, the empirical results on measuring the reconstruction errors at
the feature map give hints for the enhancements.

FEED 是一种有效的知识蒸馏（knowledge distillation）方法，旨在通过特征映射级别的集成学习，将多个教师网络的知识传递给学生网络，提高其泛化能力，同时在测试时不引入额外的参数或计算。