Knowledge distillation(KD) has demonstrated remarkable success across various
domains, but its application to medical imaging tasks, such as kidney and liver
tumor segmentation, has encountered challenges. Many existing KD methods are
not specifically tailored for these tasks. Moreover, prevalent KD methods often
lack a careful consideration of what and from where to distill knowledge from
the teacher to the student. This oversight may lead to issues like the
accumulation of training bias within shallower student layers, potentially
compromising the effectiveness of KD. To address these challenges, we propose
Hierarchical Layer-selective Feedback Distillation (HLFD). HLFD strategically
distills knowledge from a combination of middle layers to earlier layers and
transfers final layer knowledge to intermediate layers at both the feature and
pixel levels. This design allows the model to learn higher-quality
representations from earlier layers, resulting in a robust and compact student
model. Extensive quantitative evaluations reveal that HLFD outperforms existing
methods by a significant margin. For example, in the kidney segmentation task,
HLFD surpasses the student model (without KD) by over 10pp, significantly
improving its focus on tumor-specific features. From a qualitative standpoint,
the student model trained using HLFD excels at suppressing irrelevant
information and can focus sharply on tumor-specific details, which opens a new
pathway for more efficient and accurate diagnostic tools.

知识蒸馏在医学成像任务（如肾脏和肝脏肿瘤分割）中的应用面临挑战，为解决这些问题，我们提出了分层选择性反馈蒸馏（HLFD）方法，通过从中间层到较早层的蒸馏，以及将最终层的知识以特征和像素级别转移到中间层，实现模型从早期层学习更高质量的表示，从而生成一个稳健且紧凑的学生模型。大量的定量评估表明，HLFD 方法显著优于现有方法，在肾脏分割任务中，与无知识蒸馏的学生模型相比，HLFD 提升了 10 个百分点以上，显著改善了对肿瘤特定特征的关注。从质量上看，使用 HLFD 训练的学生模型能够抑制不相关信息，聚焦于肿瘤特定细节，为更高效准确的诊断工具开辟了新路径。