Fine-tuning large models is highly effective, however, inference using these
models can be expensive and produces carbon emissions. Knowledge distillation
has been shown to be a practical solution to reduce inference costs, but the
distillation process itself requires significant computational resources.
Rather than buying or renting GPUs to fine-tune, then distill a large model, an
NLP practitioner who needs a compact model might also choose to simply allocate
an available budget to hire annotators and manually label additional
fine-tuning data. In this paper, we investigate how to most efficiently use a
fixed budget to build a compact model. Through our extensive experiments on six
diverse NLP tasks, we find that distilling from T5-XXL (11B) to T5-Small (60M)
leads to almost always a cost-efficient option compared to annotating more data
to directly train a compact model (T5-Small (60M)). We further demonstrate that
the optimal amount of distillation that maximizes utility varies across
different budgetary scenarios.

通过在六个不同的 NLP 任务上进行实验，本文发现从 T5-XXL (11B) 到 T5-Small (60M) 的蒸馏通常比注释更多数据来直接训练紧凑型模型（T5-Small (60M)）更具成本效益，同时进一步证明了在不同预算情况下，最大化效用的最佳蒸馏量有所不同。

紧缩模型的经济微调：蒸馏还是注释？

Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models

Inference from large autoregressive models like Transformers is slow -
decoding K tokens takes K serial runs of the model. In this work we introduce
speculative decoding - an algorithm to sample from autoregressive models faster
without any changes to the outputs, by computing several tokens in parallel. At
the heart of our approach lie the observations that (1) hard language-modeling
tasks often include easier subtasks that can be approximated well by more
efficient models, and (2) using speculative execution and a novel sampling
method, we can make exact decoding from the large models faster, by running
them in parallel on the outputs of the approximation models, potentially
generating several tokens concurrently, and without changing the distribution.
Our method supports existing off-the-shelf models without retraining or
architecture changes. We demonstrate it on T5-XXL and show a 2X-3X acceleration
compared to the standard T5X implementation, with identical outputs.

本文提出了一种名为 “speculative decoding” 的算法，通过逐步并行地计算，采样自动回归模型可更快，同时采用了采样方法和一些新的技术，使得不改变分布的情况下，从大模型中精确解码变得更快，从而实现了不需要重新训练或架构更改即可支持现有模型的加速。在 T5-XXL 模型上的比较表明，该方法可以实现 2-3 倍的加速，而输出仍然与标准 T5X 实现相同。