We propose a new approach, Knowledge Distillation using Optimal Transport
(KNOT), to distill the natural language semantic knowledge from multiple
teacher networks to a student network. KNOT aims to train a (global) student
model by learning to minimize the optimal transport cost of its assigned
probability distribution over the labels to the weighted sum of probabilities
predicted by the (local) teacher models, under the constraints, that the
student model does not have access to teacher models' parameters or training
data. To evaluate the quality of knowledge transfer, we introduce a new metric,
Semantic Distance (SD), that measures semantic closeness between the predicted
and ground truth label distributions. The proposed method shows improvements in
the global model's SD performance over the baseline across three NLP tasks
while performing on par with Entropy-based distillation on standard accuracy
and F1 metrics. The implementation pertaining to this work is publicly
available at: this https URL

我们提出了一种新的方法，使用最优传输进行知识蒸馏 (KNOT)，从多个教师网络中提取自然语言的语义知识传输到一个学生网络中。此方法通过学习最小化分配给标签的概率分布的最优传输成本到教师模型的概率加权和的学生模型（全局）来训练学生模型，新的语义距离（SD）度量衡量预测和真实标签分布之间的语义接近程度。该方法在三个 NLP 任务上表现出了比基准更好的 SD 性能，并在标准准确性和 F1 度量上与基于熵的蒸馏表现相当。

KNOT：使用最优输运进行知识蒸馏来解决自然语言处理任务

KNOT: Knowledge Distillation using Optimal Transport for Solving NLP  Tasks

With the rapid development of deep learning, there have been an
unprecedentedly large number of trained deep network models available online.
Reusing such trained models can significantly reduce the cost of training the
new models from scratch, if not infeasible at all as the annotations used for
the training original networks are often unavailable to public. We propose in
this paper to study a new model-reusing task, which we term as \emph{knowledge
amalgamation}. Given multiple trained teacher networks, each of which
specializes in a different classification problem, the goal of knowledge
amalgamation is to learn a lightweight student model capable of handling the
comprehensive classification. We assume no other annotations except the outputs
from the teacher models are available, and thus focus on extracting and
amalgamating knowledge from the multiple teachers. To this end, we propose a
pilot two-step strategy to tackle the knowledge amalgamation task, by learning
first the compact feature representations from teachers and then the network
parameters in a layer-wise manner so as to build the student model. We apply
this approach to four public datasets and obtain very encouraging results: even
without any human annotation, the obtained student model is competent to handle
the comprehensive classification task and in most cases outperforms the
teachers in individual sub-tasks.

本研究提出了一种名为知识聚合的新型模型复用任务，旨在从多个训练好的教师网络中提取知识，学习一种能够处理综合分类的轻量级学生模型，取得了很好的结果。