Dataset distillation aims to compress a training dataset by creating a small
number of informative synthetic samples such that neural networks trained on
them perform as well as those trained on the original training dataset. Current
text dataset distillation methods create each synthetic sample as a sequence of
word embeddings instead of a text to apply gradient-based optimization;
however, such embedding-level distilled datasets cannot be used for training
other models whose word embedding weights are different from the model used for
distillation. To address this issue, we propose a novel text dataset
distillation approach, called Distilling dataset into Language Model (DiLM),
which trains a language model to generate informative synthetic training
samples as text data, instead of directly optimizing synthetic samples. We
evaluated DiLM on various text classification datasets and showed that
distilled synthetic datasets from DiLM outperform those from current coreset
selection methods. DiLM achieved remarkable generalization performance in
training different types of models and in-context learning of large language
models. Our code will be available at this https URL

通过训练一个语言模型生成信息丰富的合成训练样本作为文本数据，我们提出了一种新颖的文本数据集精炼方法，称为 Distilling dataset into Language Model (DiLM)。我们在各种文本分类数据集上评估了 DiLM，并展示了从 DiLM 中获得的合成数据集胜过当前核心集选择方法的结果。DiLM 在训练不同类型的模型和大型语言模型的上下文学习中取得了显著的泛化性能。

DiLM：文本级数据集蒸馏中的将数据集蒸馏为语言模型

DiLM: Distilling Dataset into Language Model for Text-level Dataset  Distillation

Data augmentation is a ubiquitous technique for increasing the size of
labeled training sets by leveraging task-specific data transformations that
preserve class labels. While it is often easy for domain experts to specify
individual transformations, constructing and tuning the more sophisticated
compositions typically needed to achieve state-of-the-art results is a
time-consuming manual task in practice. We propose a method for automating this
process by learning a generative sequence model over user-specified
transformation functions using a generative adversarial approach. Our method
can make use of arbitrary, non-deterministic transformation functions, is
robust to misspecified user input, and is trained on unlabeled data. The
learned transformation model can then be used to perform data augmentation for
any end discriminative model. In our experiments, we show the efficacy of our
approach on both image and text datasets, achieving improvements of 4.0
accuracy points on CIFAR-10, 1.4 F1 points on the ACE relation extraction task,
and 3.4 accuracy points when using domain-specific transformation operations on
a medical imaging dataset as compared to standard heuristic augmentation
approaches.

利用生成对抗算法学习序列模型来自动生成数据增强方法，有效提高图片和文本数据集的识别准确度。