Dataset condensation aims to condense a large dataset with a lot of training
samples into a small set. Previous methods usually condense the dataset into
the pixels format. However, it suffers from slow optimization speed and large
number of parameters to be optimized. When increasing image resolutions and
classes, the number of learnable parameters grows accordingly, prohibiting
condensation methods from scaling up to large datasets with diverse classes.
Moreover, the relations among condensed samples have been neglected and hence
the feature distribution of condensed samples is often not diverse. To solve
these problems, we propose to condense the dataset into another format, a
generative model. Such a novel format allows for the condensation of large
datasets because the size of the generative model remains relatively stable as
the number of classes or image resolution increases. Furthermore, an
intra-class and an inter-class loss are proposed to model the relation of
condensed samples. Intra-class loss aims to create more diverse samples for
each class by pushing each sample away from the others of the same class.
Meanwhile, inter-class loss increases the discriminability of samples by
widening the gap between the centers of different classes. Extensive
comparisons with state-of-the-art methods and our ablation studies confirm the
effectiveness of our method and its individual component. To our best
knowledge, we are the first to successfully conduct condensation on
ImageNet-1k.

将大型数据集通过生成模型进行压缩，通过内类和间类损失优化样本集合关系，提高优化速度和适应大型数据集的能力。

通过生成模型进行数据集简化

Dataset Condensation via Generative Model

The speed at which one can minimize an expected loss using stochastic methods
depends on two properties: the curvature of the loss and the variance of the
gradients. While most previous works focus on one or the other of these
properties, we explore how their interaction affects optimization speed.
Further, as the ultimate goal is good generalization performance, we clarify
how both curvature and noise are relevant to properly estimate the
generalization gap. Realizing that the limitations of some existing works stems
from a confusion between these matrices, we also clarify the distinction
between the Fisher matrix, the Hessian, and the covariance matrix of the
gradients.

通过研究损失曲率和梯度方差的交互作用对优化速度的影响，本文探讨了两者相互作用如何影响优化速度，并阐明了曲率和噪声对于正确估计概括性差距的相关性质，同时区分了 Fisher 矩阵、Hessian 矩阵和梯度的协方差矩阵，以澄清现有作品的局限性。