Knowledge distillation is a critical technique to transfer knowledge between
models, typically from a large model (the teacher) to a more fine-grained one
(the student). The objective function of knowledge distillation is typically
the cross-entropy between the teacher and the student's output distributions.
However, for structured prediction problems, the output space is exponential in
size; therefore, the cross-entropy objective becomes intractable to compute and
optimize directly. In this paper, we derive a factorized form of the knowledge
distillation objective for structured prediction, which is tractable for many
typical choices of the teacher and student models. In particular, we show the
tractability and empirical effectiveness of structural knowledge distillation
between sequence labeling and dependency parsing models under four different
scenarios: 1) the teacher and student share the same factorization form of the
output structure scoring function; 2) the student factorization produces more
fine-grained substructures than the teacher factorization; 3) the teacher
factorization produces more fine-grained substructures than the student
factorization; 4) the factorization forms from the teacher and the student are
incompatible.

该文提出了一种可用于解决结构化预测问题的知识蒸馏目标函数的分解形式，该分解形式对于许多师生模型的典型选择是可处理的。具体而言，它展示了四种不同情况下序列标注和依赖解析模型之间的结构知识蒸馏的可处理性和实际有效性。