A computationally expensive and memory intensive neural network lies behind
the recent success of language representation learning. Knowledge distillation,
a major technique for deploying such a vast language model in resource-scarce
environments, transfers the knowledge on individual word representations
learned without restrictions. In this paper, inspired by the recent
observations that language representations are relatively positioned and have
more semantic knowledge as a whole, we present a new knowledge distillation
objective for language representation learning that transfers the contextual
knowledge via two types of relationships across representations: Word Relation
and Layer Transforming Relation. Unlike other recent distillation techniques
for the language models, our contextual distillation does not have any
restrictions on architectural changes between teacher and student. We validate
the effectiveness of our method on challenging benchmarks of language
understanding tasks, not only in architectures of various sizes, but also in
combination with DynaBERT, the recently proposed adaptive size pruning method.

本文提出了一种新的语言表示学习的知识蒸馏方法，通过单词关系和层变换关系传递上下文知识，无需限制教师和学生的架构变化，验证了该方法在语言理解任务的各种架构和 DynaBERT 等自适应尺寸剪枝方法的挑战性基准上的有效性。