Recent works in dataset distillation seek to minimize training expenses by
generating a condensed synthetic dataset that encapsulates the information
present in a larger real dataset. These approaches ultimately aim to attain
test accuracy levels akin to those achieved by models trained on the entirety
of the original dataset. Previous studies in feature and distribution matching
have achieved significant results without incurring the costs of bi-level
optimization in the distillation process. Despite their convincing efficiency,
many of these methods suffer from marginal downstream performance improvements,
limited distillation of contextual information, and subpar cross-architecture
generalization. To address these challenges in dataset distillation, we propose
the ATtentiOn Mixer (ATOM) module to efficiently distill large datasets using a
mixture of channel and spatial-wise attention in the feature matching process.
Spatial-wise attention helps guide the learning process based on consistent
localization of classes in their respective images, allowing for distillation
from a broader receptive field. Meanwhile, channel-wise attention captures the
contextual information associated with the class itself, thus making the
synthetic image more informative for training. By integrating both types of
attention, our ATOM module demonstrates superior performance across various
computer vision datasets, including CIFAR10/100 and TinyImagenet. Notably, our
method significantly improves performance in scenarios with a low number of
images per class, thereby enhancing its potential. Furthermore, we maintain the
improvement in cross-architectures and applications such as neural architecture
search.

利用 ATtentiOn Mixer (ATOM) 模块在特征匹配过程中使用混合的通道和空间注意力，以有效地精炼大型数据集，从而在各种计算机视觉数据集中展现出卓越的性能。