Knowledge distillation is one of the most popular and effective techniques
for knowledge transfer, model compression and semi-supervised learning. Most
existing distillation approaches require the access to original or augmented
training samples. But this can be problematic in practice due to privacy,
proprietary and availability concerns. Recent work has put forward some methods
to tackle this problem, but they are either highly time-consuming or unable to
scale to large datasets. To this end, we propose a new method to train a
generative image model by leveraging the intrinsic normalization layers'
statistics of the trained teacher network. This enables us to build an ensemble
of generators without training data that can efficiently produce substitute
inputs for subsequent distillation. The proposed method pushes forward the
data-free distillation performance on CIFAR-10 and CIFAR-100 to 95.02% and
77.02% respectively. Furthermore, we are able to scale it to ImageNet dataset,
which to the best of our knowledge, has never been done using generative models
in a data-free setting.

提出了一种新的方法，通过利用训练教师网络内在归一化层的统计信息来训练生成图像模型，从而在没有训练数据的情况下实现知识迁移、模型压缩和半监督学习，该方法在 CIFAR-10 和 CIFAR-100 数据集上表现出色，并能够将其扩展到 ImageNet 数据集。

大规模生成无数据蒸馏

Large-Scale Generative Data-Free Distillation

Large pre-trained transformer-based language models have achieved impressive
results on a wide range of NLP tasks. In the past few years, Knowledge
Distillation(KD) has become a popular paradigm to compress a computationally
expensive model to a resource-efficient lightweight model. However, most KD
algorithms, especially in NLP, rely on the accessibility of the original
training dataset, which may be unavailable due to privacy issues. To tackle
this problem, we propose a novel two-stage data-free distillation method, named
Adversarial self-Supervised Data-Free Distillation (AS-DFD), which is designed
for compressing large-scale transformer-based models (e.g., BERT). To avoid
text generation in discrete space, we introduce a Plug & Play Embedding
Guessing method to craft pseudo embeddings from the teacher's hidden knowledge.
Meanwhile, with a self-supervised module to quantify the student's ability, we
adapt the difficulty of pseudo embeddings in an adversarial training manner. To
the best of our knowledge, our framework is the first data-free distillation
framework designed for NLP tasks. We verify the effectiveness of our method on
several text classification datasets.

提出一种名为 AS-DFD 的新的两阶段无数据蒸馏方法，用于压缩大型基于 Transformer 的模型（例如 BERT），并且是第一个面向 NLP 任务设计的无数据蒸馏框架，在 Text Classification 数据集上验证了其有效性。