Large language models (LLMs) are versatile and can address many tasks, but
for computational efficiency, it is often desirable to distill their
capabilities into smaller student models. One way to do this for classification
tasks is via dataset synthesis, which can be accomplished by generating
examples of each label from the LLM. Prior approaches to synthesis use few-shot
prompting, which relies on the LLM's parametric knowledge to generate usable
examples. However, this leads to issues of repetition, bias towards popular
entities, and stylistic differences from human text. In this work, we propose
Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval
augmentation to introduce variety into the dataset synthesis process: as
retrieved passages vary, the LLM is "seeded" with different content to generate
its examples. We empirically study the synthesis of six datasets, covering
topic classification, sentiment analysis, tone detection, and humor, requiring
complex synthesis strategies. We find SynthesizRR greatly improves lexical and
semantic diversity, similarity to human-written text, and distillation
performance, when compared to standard 32-shot prompting and six baseline
approaches.

通过检索和改进综合方法（SynthesizRR），我们通过扩展种子生成内容的方式来合成数据集，从而在语义和词汇多样性、与人类书写文本的相似性以及提纯效果方面极大地改善了标准 32-shot prompting 和其他六种基线方法。

SynthesizRR: 利用检索增强生成多样化数据集

SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation

Data-Free Knowledge Distillation (DFKD) plays a vital role in compressing the
model when original training data is unavailable. Previous works for DFKD in
NLP mainly focus on distilling encoder-only structures like BERT on
classification tasks, which overlook the notable progress of generative
language modeling. In this work, we propose a novel DFKD framework, namely
DFKD-T$^{3}$, where the pretrained generative language model can also serve as
a controllable data generator for model compression. This novel framework
DFKD-T$^{3}$ leads to an end-to-end learnable text-to-text framework to
transform the general domain corpus to compression-friendly task data,
targeting to improve both the \textit{specificity} and \textit{diversity}.
Extensive experiments show that our method can boost the distillation
performance in various downstream tasks such as sentiment analysis, linguistic
acceptability, and information extraction. Furthermore, we show that the
generated texts can be directly used for distilling other language models and
outperform the SOTA methods, making our method more appealing in a general DFKD
setting. Our code is available at
this https URL

提出了一种新的 DFKD 框架，即 DFKD-T^3，该框架将预训练的生成语言模型作为可控数据生成器用于模型压缩，实现了端到端可学习的文本转换框架，通过改善特异性和多样性提高了蒸馏性能，并在情感分析、语言可接受性和信息提取等各种下游任务中展示出卓越的性能，此外，生成的文本可直接用于蒸馏其他语言模型，超过了 SOTA 方法，在一般的 DFKD 场景中更具吸引力。

无数据蒸馏：通过文本到文本迁移的语言模型

Data-Free Distillation of Language Model by Text-to-Text Transfer

Conventionally, during the knowledge distillation process (e.g. feature
distillation), an additional projector is often required to perform feature
transformation due to the dimension mismatch between the teacher and the
student networks. Interestingly, we discovered that even if the student and the
teacher have the same feature dimensions, adding a projector still helps to
improve the distillation performance. In addition, projectors even improve
logit distillation if we add them to the architecture too. Inspired by these
surprising findings and the general lack of understanding of the projectors in
the knowledge distillation process from existing literature, this paper
investigates the implicit role that projectors play but so far have been
overlooked. Our empirical study shows that the student with a projector (1)
obtains a better trade-off between the training accuracy and the testing
accuracy compared to the student without a projector when it has the same
feature dimensions as the teacher, (2) better preserves its similarity to the
teacher beyond shallow and numeric resemblance, from the view of Centered
Kernel Alignment (CKA), and (3) avoids being over-confident as the teacher does
at the testing phase. Motivated by the positive effects of projectors, we
propose a projector ensemble-based feature distillation method to further
improve distillation performance. Despite the simplicity of the proposed
strategy, empirical results from the evaluation of classification tasks on
benchmark datasets demonstrate the superior classification performance of our
method on a broad range of teacher-student pairs and verify from the aspects of
CKA and model calibration that the student's features are of improved quality
with the projector ensemble design.

在知识蒸馏过程中，我们发现即使学生和老师具有相同的特征维度，添加投影器仍然有助于提高蒸馏性能，并且投影器在体系结构中的添加甚至可以改善逻辑蒸馏。受到这些令人惊讶的发现和现有文献对知识蒸馏过程中投影器角色的不了解的启发，本文研究了被忽视的投影器的隐含作用。通过经验研究，我们发现具有投影器的学生与没有投影器的学生相比，可以在训练精度和测试精度之间获得更好的平衡，从 Centered Kernel Alignment (CKA) 的视角来看，其相似性与老师相比更好地保持，并且避免了测试阶段老师的过度自信。受到投影器积极影响的启发，我们提出了一种基于投影器集合的特征蒸馏方法以进一步提高蒸馏性能。尽管所提出的策略简单，但在基准数据集上进行的分类任务的实证结果显示了我们方法在广泛范围的师生对上具有优越的分类性能，并从 CKA 和模型校准的角度验证了学生特征的质量改进。