The poor cross-architecture generalization of dataset distillation greatly
weakens its practical significance. This paper attempts to mitigate this issue
through an empirical study, which suggests that the synthetic datasets undergo
an inductive bias towards the distillation model. Therefore, the evaluation
model is strictly confined to having similar architectures of the distillation
model. We propose a novel method of EvaLuation with distillation Feature (ELF),
which utilizes features from intermediate layers of the distillation model for
the cross-architecture evaluation. In this manner, the evaluation model learns
from bias-free knowledge therefore its architecture becomes unfettered while
retaining performance. By performing extensive experiments, we successfully
prove that ELF can well enhance the cross-architecture generalization of
current DD methods. Code of this project is at
https://github.com/Lirui-Zhao/ELF.

通过实证研究，本文提出了一种新颖的评估方法 ELF，通过利用蒸馏模型的中间层特征进行跨架构评估，从无偏重的知识中学习，以提高当前 DD 方法的跨架构泛化能力。

通过实证研究提升数据集蒸馏的跨架构泛化能力

Boosting the Cross-Architecture Generalization of Dataset Distillation  through an Empirical Study

Dataset Distillation (DD) is a prominent technique that encapsulates
knowledge from a large-scale original dataset into a small synthetic dataset
for efficient training. Meanwhile, Pre-trained Models (PTMs) function as
knowledge repositories, containing extensive information from the original
dataset. This naturally raises a question: Can PTMs effectively transfer
knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct
preliminary experiments, confirming the contribution of PTMs to DD. Afterwards,
we systematically study different options in PTMs, including initialization
parameters, model architecture, training epoch and domain knowledge, revealing
that: 1) Increasing model diversity enhances the performance of synthetic
datasets; 2) Sub-optimal models can also assist in DD and outperform
well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory
for DD, but a reasonable domain match is crucial. Finally, by selecting optimal
options, we significantly improve the cross-architecture generalization over
baseline DD methods. We hope our work will facilitate researchers to develop
better DD techniques. Our code is available at
this https URL

数据集精炼（DD）是一种将大规模原始数据集的知识封装到小型合成数据集中进行高效训练的突出技术。与此同时，预训练模型（PTMs）作为知识库具有从原始数据集中获取的广泛信息。我们通过初步实验验证了 PTMs 对 DD 的贡献，然后系统研究了 PTMs 的不同选项，包括初始化参数、模型架构、训练周期和领域知识，揭示了：1）增加模型多样性可以提高合成数据集的性能；2）次优模型在某些情况下可以辅助 DD 并超过训练充分的模型；3）DD 并不一定需要特定的领域 PTMs，但合理的领域匹配至关重要。最后，通过选择最佳选项，我们显著提高了基线 DD 方法的跨体系结构泛化能力。我们希望我们的工作能帮助研究人员开发更好的 DD 技术。

预训练模型是否能在数据集精炼中提供帮助？

Can pre-trained models assist in dataset distillation?

With the rise of deep learning, large datasets and complex models have become
common, requiring significant computing power. To address this, data
distillation has emerged as a technique to quickly train models with lower
memory and time requirements. However, data distillation on text-based datasets
hasn't been explored much because of the challenges rising due to its discrete
nature. Additionally, existing dataset distillation methods often struggle to
generalize to new architectures. In the paper, we propose several data
distillation techniques for multilingual text classification datasets using
language-model-based learning methods. We conduct experiments to analyze their
performance in terms of classification strength, and cross-architecture
generalization. Furthermore, we investigate the language-specific fairness of
the data summaries generated by these methods. Our approach builds upon
existing techniques, enhancing cross-architecture generalization in the text
data distillation domain.

通过使用基于语言模型的学习方法，我们在多语言文本分类数据集上提出了几种数据蒸馏技术，来增强文本数据蒸馏领域中的跨体系结构泛化能力，并分析了它们在分类强度和跨体系结构泛化方面的性能，也研究了这些方法生成的数据摘要的语言特定公平性。

探索多语言文本数据蒸馏

Exploring Multilingual Text Data Distillation

Dataset Distillation aims to distill an entire dataset's knowledge into a few
synthetic images. The idea is to synthesize a small number of synthetic data
points that, when given to a learning algorithm as training data, result in a
model approximating one trained on the original data. Despite recent progress
in the field, existing dataset distillation methods fail to generalize to new
architectures and scale to high-resolution datasets. To overcome the above
issues, we propose to use the learned prior from pre-trained deep generative
models to synthesize the distilled data. To achieve this, we present a new
optimization algorithm that distills a large number of images into a few
intermediate feature vectors in the generative model's latent space. Our method
augments existing techniques, significantly improving cross-architecture
generalization in all settings.

Dataset Distillation technique using learned prior of deep generative models and a new optimization algorithm improves cross-architecture generalization by synthesizing few synthetic images from a large dataset.