Dataset distillation aims to minimize the time and memory needed for training
deep networks on large datasets, by creating a small set of synthetic images
that has a similar generalization performance to that of the full dataset.
However, current dataset distillation techniques fall short, showing a notable
performance gap when compared to training on the original data. In this work,
we are the first to argue that using just one synthetic subset for distillation
will not yield optimal generalization performance. This is because the training
dynamics of deep networks drastically change during the training. Hence,
multiple synthetic subsets are required to capture the training dynamics at
different phases of training. To address this issue, we propose Progressive
Dataset Distillation (PDD). PDD synthesizes multiple small sets of synthetic
images, each conditioned on the previous sets, and trains the model on the
cumulative union of these subsets without requiring additional training time.
Our extensive experiments show that PDD can effectively improve the performance
of existing dataset distillation methods by up to 4.3%. In addition, our method
for the first time enable generating considerably larger synthetic datasets.

通过使用逐步数据集提取方法，这篇研究论文提出了一种在训练期间使用多个合成子集来捕捉深度网络的训练动态，并在不增加训练时间的情况下显著改善现有数据集提取方法的性能，同时还首次实现了生成更大的合成数据集。