Dataset distillation aims to generate a smaller but representative subset
from a large dataset, which allows a model to be trained efficiently, meanwhile
evaluating on the original testing data distribution to achieve decent
performance. Many prior works have aimed to align with diverse aspects of the
original datasets, such as matching the training weight trajectories, gradient,
feature/BatchNorm distributions, etc. In this work, we show how to distill
various large-scale datasets such as full ImageNet-1K/21K under a conventional
input resolution of 224$\times$224 to achieve the best accuracy over all
previous approaches, including SRe$^2$L, TESLA and MTT. To achieve this, we
introduce a simple yet effective ${\bf C}$urriculum ${\bf D}$ata ${\bf
A}$ugmentation ($\texttt{CDA}$) during data synthesis that obtains the accuracy
on large-scale ImageNet-1K and 21K with 63.2% under IPC (Images Per Class) 50
and 36.1% under IPC 20, respectively. Finally, we show that, by integrating all
our enhancements together, the proposed model beats the current
state-of-the-art by more than 4% Top-1 accuracy on ImageNet-1K/21K and for the
first time, reduces the gap to its full-data training counterpart to less than
absolute 15%. Moreover, this work represents the inaugural success in dataset
distillation on larger-scale ImageNet-21K under the standard 224$\times$224
resolution. Our code and distilled ImageNet-21K dataset of 20 IPC, 2K recovery
budget are available at this https URL

使用课程数据增强方法，在大规模的 ImageNet-1K 和 ImageNet-21K 数据集上实现了最佳准确度，并在准确度上超越了目前的最先进模型。

大数据时代的数据集精炼

Dataset Distillation in Large Data Era

The main question we address in this paper is how to scale up visual
recognition of unseen classes, also known as zero-shot learning, to tens of
thousands of categories as in the ImageNet-21K benchmark. At this scale,
especially with many fine-grained categories included in ImageNet-21K, it is
critical to learn quality visual semantic representations that are
discriminative enough to recognize unseen classes and distinguish them from
seen ones. We propose a \emph{H}ierarchical \emph{G}raphical knowledge
\emph{R}epresentation framework for the confidence-based classification method,
dubbed as HGR-Net. Our experimental results demonstrate that HGR-Net can grasp
class inheritance relations by utilizing hierarchical conceptual knowledge. Our
method significantly outperformed all existing techniques, boosting the
performance by 7\% compared to the runner-up approach on the ImageNet-21K
benchmark. We show that HGR-Net is learning-efficient in few-shot scenarios. We
also analyzed our method on smaller datasets like ImageNet-21K-P, 2-hops and
3-hops, demonstrating its generalization ability. Our benchmark and code are
available at this https URL.

提出了一种基于层次图知识表示的置信度分类方法，命名为 HGR-Net，用于零样本学习和大规模图像识别，可以有效地识别未见过的类别并区分它们与已见过的类别，在 ImageNet-21K 基准测试中，性能比其他现有技术提升了 7%。

探索大规模零样本图像分类的分层图表示

Exploring Hierarchical Graph Representation for Large-Scale Zero-Shot Image Classification

ImageNet-1K serves as the primary dataset for pretraining deep learning
models for computer vision tasks. ImageNet-21K dataset, which is bigger and
more diverse, is used less frequently for pretraining, mainly due to its
complexity, low accessibility, and underestimation of its added value. This
paper aims to close this gap, and make high-quality efficient pretraining on
ImageNet-21K available for everyone. Via a dedicated preprocessing stage,
utilization of WordNet hierarchical structure, and a novel training scheme
called semantic softmax, we show that various models significantly benefit from
ImageNet-21K pretraining on numerous datasets and tasks, including small
mobile-oriented models. We also show that we outperform previous ImageNet-21K
pretraining schemes for prominent new models like ViT and Mixer. Our proposed
pretraining pipeline is efficient, accessible, and leads to SoTA reproducible
results, from a publicly available dataset. The training code and pretrained
models are available at: this https URL

本篇论文旨在通过专门的预处理阶段、利用 WordNet 层次结构和一种称为语义 softmax 的新型训练方案，从公开可用的 ImageNet-21K 数据集中有效地进行优质高效的预训练，并展示了不同模型在各种数据集和任务上获得的令人满意的结果。