Researchers have long tried to minimize training costs in deep learning while
maintaining strong generalization across diverse datasets. Emerging research on
dataset distillation aims to reduce training costs by creating a small
synthetic set that contains the information of a larger real dataset and
ultimately achieves test accuracy equivalent to a model trained on the whole
dataset. Unfortunately, the synthetic data generated by previous methods are
not guaranteed to distribute and discriminate as well as the original training
data, and they incur significant computational costs. Despite promising
results, there still exists a significant performance gap between models
trained on condensed synthetic sets and those trained on the whole dataset. In
this paper, we address these challenges using efficient Dataset Distillation
with Attention Matching (DataDAM), achieving state-of-the-art performance while
reducing training costs. Specifically, we learn synthetic images by matching
the spatial attention maps of real and synthetic data generated by different
layers within a family of randomly initialized neural networks. Our method
outperforms the prior methods on several datasets, including CIFAR10/100,
TinyImageNet, ImageNet-1K, and subsets of ImageNet-1K across most of the
settings, and achieves improvements of up to 6.5% and 4.1% on CIFAR100 and
ImageNet-1K, respectively. We also show that our high-quality distilled images
have practical benefits for downstream applications, such as continual learning
and neural architecture search.

采用高效的数据集提炼技术 (DataDAM)，通过匹配真实数据和合成数据的不同层级生成的空间关注图，我们在多个数据集上实现了最先进的性能同时降低了训练成本。