Dataset condensation addresses the problem of data burden by learning a small
synthetic training set that preserves essential knowledge from the larger real
training set. To date, the state-of-the-art (SOTA) results are often yielded by
optimization-oriented methods, but their inefficiency hinders their application
to realistic datasets. On the other hand, the Distribution-Matching (DM)
methods show remarkable efficiency but sub-optimal results compared to
optimization-oriented methods. In this paper, we reveal the limitations of
current DM-based methods from the inner-class and inter-class views, i.e.,
Persistent Training and Distribution Shift. To address these problems, we
propose a new DM-based method named Dual-view distribution AligNment for
dataset CondEnsation (DANCE), which exploits a few pre-trained models to
improve DM from both inner-class and inter-class views. Specifically, from the
inner-class view, we construct multiple "middle encoders" to perform pseudo
long-term distribution alignment, making the condensed set a good proxy of the
real one during the whole training process; while from the inter-class view, we
use the expert models to perform distribution calibration, ensuring the
synthetic data remains in the real class region during condensing. Experiments
demonstrate the proposed method achieves a SOTA performance while maintaining
comparable efficiency with the original DM across various scenarios. Source
codes are available at this https URL

从内部和外部视角揭示了当前基于分布匹配的方法在持久化训练和分布偏移方面的局限性，提出了一种新的基于双视图分布对齐的数据集简化方法 (DANCE)，通过利用多个预训练模型来改进分布匹配，从而取得了状态的结果，并在保持与原始分布匹配相当的效率下实现了流行的性能。

DANCE: 双视图数据集收缩的双视图分布对齐

DANCE: Dual-View Distribution Alignment for Dataset Condensation

In the era of deep learning, training deep neural networks often requires
extensive data, leading to substantial costs. Dataset condensation addresses
this by learning a small synthetic set that preserves essential information
from the original large-scale dataset. Nowadays, optimization-oriented methods
dominate dataset condensation for state-of-the-art (SOTA) results, but their
computationally intensive bi-level optimization hinders practicality with large
datasets. To enhance efficiency, as alternative solutions,
Distribution-Matching (DM)-based methods reduce costs by aligning the
representation distributions of real and synthetic examples. However, current
DM-based methods still yield less comparable results to SOTA
optimization-oriented methods. In this paper, we argue that existing DM-based
methods overlook the higher-order alignment of the distributions, which may
lead to sub-optimal matching results. Inspired by this, we propose a new
DM-based method named as Efficient Dataset Condensation by Higher-Order
Distribution Alignment (ECHO). Specifically, rather than only aligning the
first-order moment of the representation distributions as previous methods, we
learn synthetic examples via further aligning the higher-order moments of the
representation distributions of real and synthetic examples based on the
classical theory of reproducing kernel Hilbert space. Experiments demonstrate
the proposed method achieves a significant performance boost while maintaining
efficiency across various scenarios.

在深度学习时代，强调优化的数据集精简方法主导着目前最先进结果的可能性，但是它们的计算复杂度对于大规模数据集的实际应用造成了困难。为了提高效率，我们提出了一种新的基于高阶分布对⻬的分布匹配方法，通过进一步对齐真实和合成示例的表示分布的高阶矩来学习合成示例。实验证明该方法在不同场景下能够显著提升性能并保持高效。

ECHO: 高阶分布对齐的高效数据集压缩

ECHO: Efficient Dataset Condensation by Higher-Order Distribution  Alignment

Dataset Condensation aims to condense a large dataset into a smaller one
while maintaining its ability to train a well-performing model, thus reducing
the storage cost and training effort in deep learning applications. However,
conventional dataset condensation methods are optimization-oriented and
condense the dataset by performing gradient or parameter matching during model
optimization, which is computationally intensive even on small datasets and
models. In this paper, we propose a novel dataset condensation method based on
distribution matching, which is more efficient and promising. Specifically, we
identify two important shortcomings of naive distribution matching (i.e.,
imbalanced feature numbers and unvalidated embeddings for distance computation)
and address them with three novel techniques (i.e., partitioning and expansion
augmentation, efficient and enriched model sampling, and class-aware
distribution regularization). Our simple yet effective method outperforms most
previous optimization-oriented methods with much fewer computational resources,
thereby scaling data condensation to larger datasets and models. Extensive
experiments demonstrate the effectiveness of our method. Codes are available at
this https URL

通过分布匹配的方法，我们提出了一种新的数据集精简方法，有效地减少了计算资源的使用，使得数据集精简能够应用于更大的数据集和模型。