Compressing deep neural network (DNN) models becomes a very important and
necessary technique for real-world applications, such as deploying those models
on mobile devices. Knowledge distillation is one of the most popular methods
for model compression, and many studies have been made on developing this
technique. However, those studies mainly focused on classification problems,
and very few attempts have been made on regression problems, although there are
many application of DNNs on regression problems. In this paper, we propose a
new formalism of knowledge distillation for regression problems. First, we
propose a new loss function, teacher outlier rejection loss, which rejects
outliers in training samples using teacher model predictions. Second, we
consider a multi-task network with two outputs: one estimates training labels
which is in general contaminated by noisy labels; And the other estimates
teacher model's output which is expected to modify the noise labels following
the memorization effects. By considering the multi-task network, training of
the feature extraction of student models becomes more effective, and it allows
us to obtain a better student model than one trained from scratch. We performed
comprehensive evaluation with one simple toy model: sinusoidal function, and
two open datasets: MPIIGaze, and Multi-PIE. Our results show consistent
improvement in accuracy regardless of the annotation error level in the
datasets.

本文提出了一种用于回归问题的新的知识蒸馏形式，其中包括教师异常值拒绝损失函数和多任务网络等内容，并通过对多个数据集的评估表明，该方法可以提高准确性。