Recent advancements in dataset distillation have demonstrated the significant
benefits of employing soft labels generated by pre-trained teacher models. In
this paper, we introduce a novel perspective by emphasizing the full
utilization of labels. We first conduct a comprehensive comparison of various
loss functions for soft label utilization in dataset distillation, revealing
that the model trained on the synthetic dataset exhibits high sensitivity to
the choice of loss function for soft label utilization. This finding highlights
the necessity of a universal loss function for training models on synthetic
datasets. Building on these insights, we introduce an extremely simple yet
surprisingly effective plug-and-play approach, GIFT, which encompasses soft
label refinement and a cosine similarity-based loss function to efficiently
leverage full label information. Extensive experiments demonstrate that GIFT
consistently enhances the state-of-the-art dataset distillation methods across
various scales datasets without incurring additional computational costs. For
instance, on ImageNet-1K with IPC = 10, GIFT improves the SOTA method RDED by
3.9% and 1.8% on ConvNet and ResNet-18, respectively. Code:
this https URL

近期关于数据集蒸馏的研究表明，采用预训练教师模型生成的软标签具有显著优势。本文从一个新的角度强调标签的充分利用。我们首先对用于数据集蒸馏的软标签利用的各种损失函数进行了全面比较，发现在合成数据集上训练的模型对于损失函数的选择具有很高的敏感性。这一发现凸显了在合成数据集上训练模型所需的通用损失函数的必要性。基于这些认识，我们引入了一种极其简单却出奇有效的即插即用方法 GIFT，它包括软标签的优化和基于余弦相似性的损失函数，以有效利用全面的标签信息。大量的实验证明，GIFT 方法始终提升了各种规模的数据集蒸馏方法的最新水平，而不增加额外的计算成本。例如，在 ImageNet-1K 上，当 IPC = 10 时，GIFT 在 ConvNet 和 ResNet-18 上分别将 SOTA 方法 RDED 的性能提升了 3.9% 和 1.8%。