Knowledge distillation (KD) is an efficient approach to transfer the
knowledge from a large "teacher" network to a smaller "student" network.
Traditional KD methods require lots of labeled training samples and a white-box
teacher (parameters are accessible) to train a good student. However, these
resources are not always available in real-world applications. The distillation
process often happens at an external party side where we do not have access to
much data, and the teacher does not disclose its parameters due to security and
privacy concerns. To overcome these challenges, we propose a black-box few-shot
KD method to train the student with few unlabeled training samples and a
black-box teacher. Our main idea is to expand the training set by generating a
diverse set of out-of-distribution synthetic images using MixUp and a
conditional variational auto-encoder. These synthetic images along with their
labels obtained from the teacher are used to train the student. We conduct
extensive experiments to show that our method significantly outperforms recent
SOTA few/zero-shot KD methods on image classification tasks. The code and
models are available at: this https URL

本文提出了一种黑盒少样本知识蒸馏方法，使用 MixUp 和条件变分自编码器生成多样的合成图像进行训练，显著优于最新 SOTA 的少 / 零样本 KD 方法用于图像分类任务。

黑盒少样本知识蒸馏

Black-box Few-shot Knowledge Distillation

Knowledge distillation (KD) is a successful approach for deep neural network
acceleration, with which a compact network (student) is trained by mimicking
the softmax output of a pre-trained high-capacity network (teacher). In
tradition, KD usually relies on access to the training samples and the
parameters of the white-box teacher to acquire the transferred knowledge.
However, these prerequisites are not always realistic due to storage costs or
privacy issues in real-world applications. Here we propose the concept of
decision-based black-box (DB3) knowledge distillation, with which the student
is trained by distilling the knowledge from a black-box teacher (parameters are
not accessible) that only returns classes rather than softmax outputs. We start
with the scenario when the training set is accessible. We represent a sample's
robustness against other classes by computing its distances to the teacher's
decision boundaries and use it to construct the soft label for each training
sample. After that, the student can be trained via standard KD. We then extend
this approach to a more challenging scenario in which even accessing the
training data is not feasible. We propose to generate pseudo samples
distinguished by the teacher's decision boundaries to the largest extent and
construct soft labels for them, which are used as the transfer set. We evaluate
our approaches on various benchmark networks and datasets and experiment
results demonstrate their effectiveness. Codes are available at:
this https URL

该研究提出一种基于决策边界的黑盒知识蒸馏方法（DB3），包括由教师决策边界构造软标签，以及生成伪样本以代表未知数据，皆不需要接触教师参数，结果表明该方法在各种基准网络和数据集上都表现出有效性。