Knowledge distillation (KD) emerges as a challenging yet promising technique
for compressing deep learning models, characterized by the transmission of
extensive learning representations from proficient and computationally
intensive teacher models to compact student models. However, only a handful of
studies have endeavored to compress the models for single image
super-resolution (SISR) through KD, with their effects on student model
enhancement remaining marginal. In this paper, we put forth an approach from
the perspective of efficient data utilization, namely, the Data Upcycling
Knowledge Distillation (DUKD) which facilitates the student model by the prior
knowledge teacher provided via upcycled in-domain data derived from their
inputs. This upcycling process is realized through two efficient image zooming
operations and invertible data augmentations which introduce the label
consistency regularization to the field of KD for SISR and substantially boosts
student model's generalization. The DUKD, due to its versatility, can be
applied across a broad spectrum of teacher-student architectures. Comprehensive
experiments across diverse benchmarks demonstrate that our proposed DUKD method
significantly outperforms previous art, exemplified by an increase of up to
0.5dB in PSNR over baselines methods, and a 67% parameters reduced RCAN model's
performance remaining on par with that of the RCAN teacher model.

通过数据再利用的知识蒸馏技术（DUKD）实现对单幅图像超分辨率的模型压缩，通过引入标签一致性规范化，大幅提升学生模型的泛化性能，相比基线方法在 PSNR 上提升了 0.5dB，并且在参数减少 67% 的 RCAN 模型中保持与 RCAN 教师模型相当的性能。