Dataset distillation aims to generate a smaller but representative subset
from a large dataset, which allows a model to be trained efficiently, meanwhile
evaluating on the original testing data distribution to achieve decent
performance. Many prior works have aimed to align with diverse aspects of the
original datasets, such as matching the training weight trajectories, gradient,
feature/BatchNorm distributions, etc. In this work, we show how to distill
various large-scale datasets such as full ImageNet-1K/21K under a conventional
input resolution of 224$\times$224 to achieve the best accuracy over all
previous approaches, including SRe$^2$L, TESLA and MTT. To achieve this, we
introduce a simple yet effective ${\bf C}$urriculum ${\bf D}$ata ${\bf
A}$ugmentation ($\texttt{CDA}$) during data synthesis that obtains the accuracy
on large-scale ImageNet-1K and 21K with 63.2% under IPC (Images Per Class) 50
and 36.1% under IPC 20, respectively. Finally, we show that, by integrating all
our enhancements together, the proposed model beats the current
state-of-the-art by more than 4% Top-1 accuracy on ImageNet-1K/21K and for the
first time, reduces the gap to its full-data training counterpart to less than
absolute 15%. Moreover, this work represents the inaugural success in dataset
distillation on larger-scale ImageNet-21K under the standard 224$\times$224
resolution. Our code and distilled ImageNet-21K dataset of 20 IPC, 2K recovery
budget are available at this https URL

使用课程数据增强方法，在大规模的 ImageNet-1K 和 ImageNet-21K 数据集上实现了最佳准确度，并在准确度上超越了目前的最先进模型。

大数据时代的数据集精炼

Dataset Distillation in Large Data Era

Curriculum Data Augmentation (CDA) improves neural models by presenting
synthetic data with increasing difficulties from easy to hard. However,
traditional CDA simply treats the ratio of word perturbation as the difficulty
measure and goes through the curriculums only once. This paper presents
\textbf{PCC}: \textbf{P}araphrasing with Bottom-k Sampling and \textbf{C}yclic
Learning for \textbf{C}urriculum Data Augmentation, a novel CDA framework via
paraphrasing, which exploits the textual paraphrase similarity as the
curriculum difficulty measure. We propose a curriculum-aware paraphrase
generation module composed of three units: a paraphrase candidate generator
with bottom-k sampling, a filtering mechanism and a difficulty measure. We also
propose a cyclic learning strategy that passes through the curriculums multiple
times. The bottom-k sampling is proposed to generate super-hard instances for
the later curriculums. Experimental results on few-shot text classification as
well as dialogue generation indicate that PCC surpasses competitive baselines.
Human evaluation and extensive case studies indicate that bottom-k sampling
effectively generates super-hard instances, and PCC significantly improves the
baseline dialogue agent.

本文提出了一种用于课程数据增强（CDA）的新框架 PCC：利用底部 k 采样的改写和循环学习的课程数据增强，通过改写利用文本改写相似性作为课程难度度量标准，并建议通过循环学习策略多次通过课程。在少样本文本分类和对话生成方面的实验结果表明，PCC 优于竞争基线。

使用底部 - k 抽样和周期学习进行课程数据增强的释义

PCC: Paraphrasing with Bottom-k Sampling and Cyclic Learning for Curriculum Data Augmentation

Few-shot text classification is a fundamental NLP task in which a model aims
to classify text into a large number of categories, given only a few training
examples per category. This paper explores data augmentation -- a technique
particularly suitable for training with limited data -- for this few-shot,
highly-multiclass text classification setting. On four diverse text
classification tasks, we find that common data augmentation techniques can
improve the performance of triplet networks by up to 3.0% on average.
To further boost performance, we present a simple training strategy called
curriculum data augmentation, which leverages curriculum learning by first
training on only original examples and then introducing augmented data as
training progresses. We explore a two-stage and a gradual schedule, and find
that, compared with standard single-stage training, curriculum data
augmentation trains faster, improves performance, and remains robust to high
amounts of noising from augmentation.

本文研究采用数据增强的方式以应对有限数据的训练，探索其中的课程数据增强方法。研究结果表明，对于少样本、多分类文本分类任务，常见的数据增强技术能显著提高三元组网络的性能，而采取课程数据增强的方式可以进一步提升模型性能，并且可以保持对于数据扰动的鲁棒性。