Knowledge distillation (KD) is the process of transferring knowledge from a
large model to a small one. It has gained increasing attention in the natural
language processing community, driven by the demands of compressing
ever-growing language models. In this work, we propose an f-DISTILL framework,
which formulates sequence-level knowledge distillation as minimizing a
generalized f-divergence function. We propose four distilling variants under
our framework and show that existing SeqKD and ENGINE approaches are
approximations of our f-DISTILL methods. We further derive step-wise
decomposition for our f-DISTILL, reducing intractable sequence-level divergence
to word-level losses that can be computed in a tractable manner. Experiments
across four datasets show that our methods outperform existing KD approaches,
and that our symmetric distilling losses can better force the student to learn
from the teacher distribution.

我们提出了一个名为 f-DISTILL 的框架，将序列级别的知识蒸馏作为最小化广义 f - 分歧函数的过程，并显示现有的 SeqKD 和 ENGINE 方法是我们 f-DISTILL 方法的近似。通过在四个数据集上的实验，我们发现我们的方法优于现有的蒸馏方法，并且我们的对称蒸馏损失可以更好地促使学生从教师分布中学习。