Typical technique in knowledge distillation (KD) is regularizing the learning
of a limited capacity model (student) by pushing its responses to match a
powerful model's (teacher). Albeit useful especially in the penultimate layer
and beyond, its action on student's feature transform is rather implicit,
limiting its practice in the intermediate layers. To explicitly embed the
teacher's knowledge in feature transform, we propose a learnable KD layer for
the student which improves KD with two distinct abilities: i) learning how to
leverage the teacher's knowledge, enabling to discard nuisance information, and
ii) feeding forward the transferred knowledge deeper. Thus, the student enjoys
the teacher's knowledge during the inference besides training. Formally, we
repurpose 1x1-BN-ReLU-1x1 convolution block to assign a semantic vector to each
local region according to the template (supervised by the teacher) that the
corresponding region of the student matches. To facilitate template learning in
the intermediate layers, we propose a novel form of supervision based on the
teacher's decisions. Through rigorous experimentation, we demonstrate the
effectiveness of our approach on 3 popular classification benchmarks. Code is
available at: this https URL

通过引入可学习的 KD 层和模板学习方法，我们提出一种新的知识蒸馏技术，实现了对学生模型在中间层中进行特征变换的显式控制，并在多个分类基准测试中验证了其有效性。

让学生决策的知识蒸馏层

Knowledge Distillation Layer that Lets the Student Decide

How to generate descriptions from structured data organized in tables?
Existing approaches using neural encoder-decoder models often suffer from
lacking diversity. We claim that an open set of templates is crucial for
enriching the phrase constructions and realizing varied generations. Learning
such templates is prohibitive since it often requires a large paired <table,
description> corpus, which is seldom available. This paper explores the problem
of automatically learning reusable "templates" from paired and non-paired data.
We propose the variational template machine (VTM), a novel method to generate
text descriptions from data tables. Our contributions include: a) we carefully
devise a specific model architecture and losses to explicitly disentangle text
template and semantic content information, in the latent spaces, and b)we
utilize both small parallel data and large raw text without aligned tables to
enrich the template learning. Experiments on datasets from a variety of
different domains show that VTM is able to generate more diversely while
keeping a good fluency and quality.

本文提出了一种用于从结构化数据生成文本描述的新方法 - VTM (变分模板机)，该方法能够从有配对和无配对的数据中学习生成模板，提高文本生成的多样性并保持良好的流畅度和质量。