We study the problem of dataset distillation - creating a small set of
synthetic examples capable of training a good model. In particular, we study
the problem of label distillation - creating synthetic labels for a small set
of real images, and show it to be more effective than the prior image-based
approach to dataset distillation. Methodologically, we introduce a more robust
and flexible meta-learning algorithm for distillation, as well as an effective
first-order strategy based on convex optimization layers. Distilling labels
with our new algorithm leads to improved results over prior image-based
distillation. More importantly, it leads to clear improvements in flexibility
of the distilled dataset in terms of compatibility with off-the-shelf
optimizers and diverse neural architectures. Interestingly, label distillation
can also be applied across datasets, for example enabling learning Japanese
character recognition by training only on synthetically labeled English
letters.

针对数据集蒸馏的问题，我们提出了用合成标签来训练模型，比基于图像的方法更为有效；我们引入了更加鲁棒和灵活的元学习算法以及一种基于凸优化层的一阶策略，这种新算法可以提高模型的性能，并且可兼容各个优化器及不同的神经结构。我们的研究发现，标签蒸馏还能夸数据集应用，例如只通过合成标签的英文字母来训练以学习日文字母识别。

数据集蒸馏：学习标签而非图像

Flexible Dataset Distillation: Learn Labels Instead of Images

The comparative losses (typically, triplet loss) are appealing choices for
learning person re-identification (ReID) features. However, the triplet loss is
computationally much more expensive than the (practically more popular)
classification loss, limiting their wider usage in massive datasets. Moreover,
the abundance of label noise and outliers in ReID datasets may also put the
margin-based loss in jeopardy. This work addresses the above two shortcomings
of triplet loss, extending its effectiveness to large-scale ReID datasets with
potentially noisy labels. We propose a fast-approximated triplet (FAT) loss,
which provably converts the point-wise triplet loss into its upper bound form,
consisting of a point-to-set loss term plus cluster compactness regularization.
It preserves the effectiveness of triplet loss, while leading to linear
complexity to the training set size. A label distillation strategy is further
designed to learn refined soft-labels in place of the potentially noisy labels,
from only an identified subset of confident examples, through teacher-student
networks. We conduct extensive experiments on three most popular ReID
benchmarks (Market-1501, DukeMTMC-reID, and MSMT17), and demonstrate that FAT
loss with distilled labels lead to ReID features with remarkable accuracy,
efficiency, robustness, and direct transferability to unseen datasets.

提出了一种快速逼近的三元组损失（FAT）来匹配人物重新识别任务的大规模数据集中的潜在噪声标签，同时提出了标签蒸馏策略来改进由于潜在噪声标签的学习，结果表明该方法可显著提高 ReID 特征的准确性，效率，鲁棒性和直接可迁移性。