Teacher-student knowledge distillation is a popular technique for compressing
today's prevailing large language models into manageable sizes that fit
low-latency downstream applications. Both the teacher and the choice of
transfer set used for distillation are crucial ingredients in creating a high
quality student. Yet, the generic corpora used to pretrain the teacher and the
corpora associated with the downstream target domain are often significantly
different, which raises a natural question: should the student be distilled
over the generic corpora, so as to learn from high-quality teacher predictions,
or over the downstream task corpora to align with finetuning? Our study
investigates this trade-off using Domain Classification (DC) and Intent
Classification/Named Entity Recognition (ICNER) as downstream tasks. We distill
several multilingual students from a larger multilingual LM with varying
proportions of generic and task-specific datasets, and report their performance
after finetuning on DC and ICNER. We observe significant improvements across
tasks and test sets when only task-specific corpora is used. We also report on
how the impact of adding task-specific data to the transfer set correlates with
the similarity between generic and task-specific data. Our results clearly
indicate that, while distillation from a generic LM benefits downstream tasks,
students learn better using target domain data even if it comes at the price of
noisier teacher predictions. In other words, target domain data still trumps
teacher knowledge.

本文研究了采用不同数据集训练的应用领域教师和泛化语言模型辅导学生学习的关系，并发现即使存在较高噪声，使用应用领域数据进行辅导仍然比使用泛化数据表现更好。