Deep neural network (DNN) models for retinopathy have estimated predictive
accuracies in the mid-to-high 90%. However, the following aspects remain
unaddressed: State-of-the-art models are complex and require substantial
computational infrastructure to train and deploy; The reliability of
predictions can vary widely. In this paper, we focus on these aspects and
propose a form of iterative knowledge distillation(IKD), called IKD+ that
incorporates a tradeoff between size, accuracy and reliability. We investigate
the functioning of IKD+ using two widely used techniques for estimating model
calibration (Platt-scaling and temperature-scaling), using the best-performing
model available, which is an ensemble of EfficientNets with approximately 100M
parameters. We demonstrate that IKD+ equipped with temperature-scaling results
in models that show up to approximately 500-fold decreases in the number of
parameters than the original ensemble without a significant loss in accuracy.
In addition, calibration scores (reliability) for the IKD+ models are as good
as or better than the base mode

本文提出了一种迭代知识蒸馏 (IKD+) 方法，它平衡了模型大小、预测准确性和可靠性，并使用两种常用的模型校准技术对其进行了验证，结果表明在保持准确性的同时，IKD + 方法的模型参数量可以减少近 500 倍。在可靠性方面，IKD + 方法的校准得分与基准模型相当甚至更好。