Deploying large language models (LLMs) of several billion parameters can be
impractical in most industrial use cases due to constraints such as cost,
latency limitations, and hardware accessibility. Knowledge distillation (KD)
offers a solution by compressing knowledge from resource-intensive large models
to smaller ones. Various strategies exist, some relying on the text generated
by the teacher model and optionally utilizing his logits to enhance learning.
However, these methods based on logits often require both teacher and student
models to share the same tokenizer, limiting their applicability across
different LLM families. In this paper, we introduce Universal Logit
Distillation (ULD) loss, grounded in optimal transport, to address this
limitation. Our experimental results demonstrate the effectiveness of ULD loss
in enabling distillation across models with different architectures and
tokenizers, paving the way to a more widespread use of distillation techniques.

通过使用基于最优传输的通用对数蒸馏（ULD）损失函数，我们解决了传统方法中教师模型和学生模型需共享分词器的限制，提供了一种跨不同架构和分词器进行蒸馏的有效方法，为蒸馏技术的广泛应用铺平了道路。