Knowledge distillation is an approach to transfer information on
representations from a teacher to a student by reducing their difference. A
challenge of this approach is to reduce the flexibility of the student's
representations inducing inaccurate learning of the teacher's knowledge. To
resolve it in transferring, we investigate distillation of structures of
representations specified to three types: intra-feature, local inter-feature,
global inter-feature structures. To transfer them, we introduce feature
structure distillation methods based on the Centered Kernel Alignment, which
assigns a consistent value to similar features structures and reveals more
informative relations. In particular, a memory-augmented transfer method with
clustering is implemented for the global structures. The methods are
empirically analyzed on the nine tasks for language understanding of the GLUE
dataset with Bidirectional Encoder Representations from Transformers (BERT),
which is a representative neural language model. In the results, the proposed
methods effectively transfer the three types of structures and improve
performance compared to state-of-the-art distillation methods. Indeed, the code
for the methods is available in this https URL.

研究了应用基于中心核对齐的特征结构蒸馏方法来有效转移三种类型的表示结构以提高 GLUE 数据集上 Bidirectional Encoder Representations from Transformers (BERT) 任务的表现，结果表明所提出的方法可以提高性能。