Currently, the reduction in the parameter scale of large-scale pre-trained
language models (PLMs) through knowledge distillation has greatly facilitated
their widespread deployment on various devices. However, the deployment of
knowledge distillation systems faces great challenges in real-world
industrial-strength applications, which require the use of complex distillation
methods on even larger-scale PLMs (over 10B), limited by memory on GPUs and the
switching of methods. To overcome these challenges, we propose GKD, a general
knowledge distillation framework that supports distillation on larger-scale
PLMs using various distillation methods. With GKD, developers can build larger
distillation models on memory-limited GPUs and easily switch and combine
different distillation methods within a single framework. Experimental results
show that GKD can support the distillation of at least 100B-scale PLMs and 25
mainstream methods on 8 NVIDIA A100 (40GB) GPUs.

提出 GKD 框架，支持在内存受限的 GPU 上使用各种蒸馏方法，以支持至少 100B 规模的 PLMs 的蒸馏，可在单个框架内轻松切换和组合不同的蒸馏方法。