Knowledge distillation (KD) methods compress large models into smaller
students with manually-designed student architectures given pre-specified
computational cost. This requires several trials to find a viable student, and
further repeating the process for each student or computational budget change.
We use Neural Architecture Search (NAS) to automatically distill several
compressed students with variable cost from a large model. Current works train
a single SuperLM consisting of millions of subnetworks with weight-sharing,
resulting in interference between subnetworks of different sizes. Our framework
AutoDistil addresses above challenges with the following steps: (a)
Incorporates inductive bias and heuristics to partition Transformer search
space into K compact sub-spaces (K=3 for typical student sizes of base, small
and tiny); (b) Trains one SuperLM for each sub-space using task-agnostic
objective (e.g., self-attention distillation) with weight-sharing of students;
(c) Lightweight search for the optimal student without re-training. Fully
task-agnostic training and search allow students to be reused for fine-tuning
on any downstream task. Experiments on GLUE benchmark against state-of-the-art
KD and NAS methods demonstrate AutoDistil to outperform leading compression
techniques with upto 2.7x reduction in computational cost and negligible loss
in task performance.

该篇论文提出了一种名为 AutoDistil 的新 KD 方法，利用 NAS 自动蒸馏出可变成本的压缩模型，该方法使用归纳偏差和技巧将搜索空间分成 K 个紧凑的子空间，并使用每个子空间产生的超级模型进行训练，以在不需要重新训练的情况下进行轻量级搜索，实验表明该方法在 GLUE 基准测试中表现优异。