Finetuning pretrained language models (LMs) have enabled appealing
performance on a diverse array of tasks. The intriguing task-agnostic property
has driven a shifted focus from task-specific to task-agnostic distillation of
LMs. While task-agnostic, compute-efficient, performance-preserved LMs can be
yielded by task-agnostic distillation, previous studies mainly sit in
distillation of either encoder-only LMs (e.g., BERT) or decoder-only ones
(e.g., GPT) yet largely neglect that distillation of encoder-decoder LMs (e.g.,
T5) can posit very distinguished behaviors. Frustratingly, we discover that
existing task-agnostic distillation methods can fail to handle the distillation
of encoder-decoder LMs. To the demand, we explore a few paths and uncover a
path named as MiniEnD that successfully tackles the distillation of
encoder-decoder LMs in a task-agnostic fashion. We examine MiniEnD on language
understanding and abstractive summarization. The results showcase that MiniEnD
is generally effective and is competitive compared to other alternatives. We
further scale MiniEnD up to distillation of 3B encoder-decoder language models
with interpolated distillation. The results imply the opportunities and
challenges in distilling large language models (e.g., LLaMA).

通过探讨 MiniEnD 方法来针对语言模型（LMs）的无特定任务的精炼，以及针对编码 - 解码 LMs 的精炼，旨在让它们保持性能同时实现计算有效，从而在语言理解和摘要制作方面得到好的效果，而且 MiniEnD 与其他备选方案相比具有良好的竞争力，同时还探索了扩展 MiniEnD 到 3B 编码器 - 解码器语言模型的跨越式提升。