Most dataset distillation methods struggle to accommodate large-scale
datasets due to their substantial computational and memory requirements. In
this paper, we present a curriculum-based dataset distillation framework
designed to harmonize scalability with efficiency. This framework strategically
distills synthetic images, adhering to a curriculum that transitions from
simple to complex. By incorporating curriculum evaluation, we address the issue
of previous methods generating images that tend to be homogeneous and
simplistic, doing so at a manageable computational cost. Furthermore, we
introduce adversarial optimization towards synthetic images to further improve
their representativeness and safeguard against their overfitting to the neural
network involved in distilling. This enhances the generalization capability of
the distilled images across various neural network architectures and also
increases their robustness to noise. Extensive experiments demonstrate that our
framework sets new benchmarks in large-scale dataset distillation, achieving
substantial improvements of 11.1\% on Tiny-ImageNet, 9.0\% on ImageNet-1K, and
7.3\% on ImageNet-21K. The source code will be released to the community.

本论文提出了一个基于课程设置的数据集蒸馏框架，通过将从简单到复杂的课程进行逐步蒸馏，合理处理合成图像的生成和评估，进一步利用对抗优化改进图像的代表性，提高其在不同神经网络体系结构下的泛化能力和抗噪性，实现了大规模数据集蒸馏的新突破。

课程数据蒸馏

Curriculum Dataset Distillation

Most existing distillation methods ignore the flexible role of the
temperature in the loss function and fix it as a hyper-parameter that can be
decided by an inefficient grid search. In general, the temperature controls the
discrepancy between two distributions and can faithfully determine the
difficulty level of the distillation task. Keeping a constant temperature,
i.e., a fixed level of task difficulty, is usually sub-optimal for a growing
student during its progressive learning stages. In this paper, we propose a
simple curriculum-based technique, termed Curriculum Temperature for Knowledge
Distillation (CTKD), which controls the task difficulty level during the
student's learning career through a dynamic and learnable temperature.
Specifically, following an easy-to-hard curriculum, we gradually increase the
distillation loss w.r.t. the temperature, leading to increased distillation
difficulty in an adversarial manner. As an easy-to-use plug-in technique, CTKD
can be seamlessly integrated into existing knowledge distillation frameworks
and brings general improvements at a negligible additional computation cost.
Extensive experiments on CIFAR-100, ImageNet-2012, and MS-COCO demonstrate the
effectiveness of our method. Our code is available at
this https URL

本文提出了一种名为 CTKD 的简单课程温度知识蒸馏技术，通过动态可学习的温度控制任务难度水平，实现了根据学生学习阶段逐步提高知识蒸馏难度的功能，并在 CIFAR-100，ImageNet-2012 和 MS-COCO 上进行了广泛的实验，证明了这种方法的有效性。

知识蒸馏的课程温度

Curriculum Temperature for Knowledge Distillation

Polar codes are widely used state-of-the-art codes for reliable communication
that have recently been included in the 5th generation wireless standards (5G).
However, there remains room for the design of polar decoders that are both
efficient and reliable in the short blocklength regime. Motivated by recent
successes of data-driven channel decoders, we introduce a novel
$\textbf{C}$ur$\textbf{RI}$culum based $\textbf{S}$equential neural decoder for
$\textbf{P}$olar codes (CRISP). We design a principled curriculum, guided by
information-theoretic insights, to train CRISP and show that it outperforms the
successive-cancellation (SC) decoder and attains near-optimal reliability
performance on the Polar(32,16) and Polar(64,22) codes. The choice of the
proposed curriculum is critical in achieving the accuracy gains of CRISP, as we
show by comparing against other curricula. More notably, CRISP can be readily
extended to Polarization-Adjusted-Convolutional (PAC) codes, where existing SC
decoders are significantly less reliable. To the best of our knowledge, CRISP
constructs the first data-driven decoder for PAC codes and attains near-optimal
performance on the PAC(32,16) code.

本文提出一种基于课程的顺序神经网络解码器（CRISP）用于极化码，该解码器基于信息论指导下的有原则的课程设计，使用数据驱动策略训练网络，通过与其他课程的比较，CRISP 在 Polar（32,16）和 Polar（64,22）码上表现出色，在现有 SC 解码器可靠性较低的情况下，CRISP 可轻松扩展到极化调整卷积（PAC）码，并且成为第一个在 PAC（32,16）码上表现近乎最优的数据驱动解码器。