Diffusion models excel at producing high-quality samples but naively require
hundreds of iterations, prompting multiple attempts to distill the generation
process into a faster network. However, many existing approaches suffer from a
variety of challenges: the process for distillation training can be complex,
often requiring multiple training stages, and the resulting models perform
poorly when utilized in single-step generative applications. In this paper, we
introduce a simple yet effective means of distilling diffusion models directly
from initial noise to the resulting image. Of particular importance to our
approach is to leverage a new Deep Equilibrium (DEQ) model as the distilled
architecture: the Generative Equilibrium Transformer (GET). Our method enables
fully offline training with just noise/image pairs from the diffusion model
while achieving superior performance compared to existing one-step methods on
comparable training budgets. We demonstrate that the DEQ architecture is
crucial to this capability, as GET matches a $5\times$ larger ViT in terms of
FID scores while striking a critical balance of computational cost and image
quality. Code, checkpoints, and datasets are available.

本论文介绍了一种简单但有效的方法，即将扩散模型直接从初始噪声提炼到生成的图像，使用深度平衡模型作为提炼架构。该方法通过仅使用扩散模型中的噪声 / 图像对进行离线训练，在与现有一步方法相当的训练预算上实现了卓越的性能。

基于深度平衡模型的一步扩散蒸馏

One-Step Diffusion Distillation via Deep Equilibrium Models

Training-free Vision Transformer (ViT) architecture search is presented to
search for a better ViT with zero-cost proxies. While ViTs achieve significant
distillation gains from CNN teacher models on small datasets, the current
zero-cost proxies in ViTs do not generalize well to the distillation training
paradigm according to our experimental observations. In this paper, for the
first time, we investigate how to search in a training-free manner with the
help of teacher models and devise an effective Training-free ViT (TVT) search
framework. Firstly, we observe that the similarity of attention maps between
ViT and ConvNet teachers affects distill accuracy notably. Thus, we present a
teacher-aware metric conditioned on the feature attention relations between
teacher and student. Additionally, TVT employs the L2-Norm of the student's
weights as the student-capability metric to improve ranking consistency.
Finally, TVT searches for the best ViT for distilling with ConvNet teachers via
our teacher-aware metric and student-capability metric, resulting in impressive
gains in efficiency and effectiveness. Extensive experiments on various tiny
datasets and search spaces show that our TVT outperforms state-of-the-art
training-free search methods. The code will be released.

本文提出了一种无需训练的视觉 Transformer（ViT）架构搜索方法，通过使用教师模型，利用具有零成本代理的方式搜索更好的 ViT，并在各种小型数据集和搜索空间上展开了大量实验，证明了该方法优于现有的无训练搜索方法。

TVT: 在小数据集上无需训练的视觉 Transformer 搜索

TVT: Training-Free Vision Transformer Search on Tiny Datasets

Neural networks trained on distilled data often produce over-confident output
and require correction by calibration methods. Existing calibration methods
such as temperature scaling and mixup work well for networks trained on
original large-scale data. However, we find that these methods fail to
calibrate networks trained on data distilled from large source datasets. In
this paper, we show that distilled data lead to networks that are not
calibratable due to (i) a more concentrated distribution of the maximum logits
and (ii) the loss of information that is semantically meaningful but unrelated
to classification tasks. To address this problem, we propose Masked Temperature
Scaling (MTS) and Masked Distillation Training (MDT) which mitigate the
limitations of distilled data and achieve better calibration results while
maintaining the efficiency of dataset distillation.

神经网络在蒸馏数据上训练往往会产生过于自信的输出，需要通过温度标定和混合等校准方法进行修正。然而，现有的校准方法对于在大型源数据蒸馏产生的网络并不适用。本文显示，由于最大逻辑概率的分布更集中以及语义有意义但与分类任务无关的信息的丢失，蒸馏数据导致的网络不可校准。为了解决该问题，我们提出了掩码温度标定（MTS）和掩码蒸馏训练（MDT），可以缓解蒸馏数据的限制，实现更好的校准结果，并保持数据集蒸馏的效率。