Knowledge distillation extracts general knowledge from a pre-trained teacher
network and provides guidance to a target student network. Most studies
manually tie intermediate features of the teacher and student, and transfer
knowledge through pre-defined links. However, manual selection often constructs
ineffective links that limit the improvement from the distillation. There has
been an attempt to address the problem, but it is still challenging to identify
effective links under practical scenarios. In this paper, we introduce an
effective and efficient feature distillation method utilizing all the feature
levels of the teacher without manually selecting the links. Specifically, our
method utilizes an attention-based meta-network that learns relative
similarities between features, and applies identified similarities to control
distillation intensities of all possible pairs. As a result, our method
determines competent links more efficiently than the previous approach and
provides better performance on model compression and transfer learning tasks.
Further qualitative analyses and ablative studies describe how our method
contributes to better distillation. The implementation code is available at
github.com/clovaai/attention-feature-distillation.

本文提出一种利用基于注意力的元网络来实现特征蒸馏的方法，该方法在不手动选择链接的情况下有效地控制了所有可能的特征对的蒸馏强度，从而提供了更好的模型压缩和迁移学习任务的性能。

基于注意力特征匹配的知识蒸馏

Show, Attend and Distill:Knowledge Distillation via Attention-based  Feature Matching

Convolutional neural networks (CNNs) have recently emerged as a popular
building block for natural language processing (NLP). Despite their success,
most existing CNN models employed in NLP share the same learned (and static)
set of filters for all input sentences. In this paper, we consider an approach
of using a small meta network to learn context-sensitive convolutional filters
for text processing. The role of meta network is to abstract the contextual
information of a sentence or document into a set of input-aware filters. We
further generalize this framework to model sentence pairs, where a
bidirectional filter generation mechanism is introduced to encapsulate
co-dependent sentence representations. In our benchmarks on four different
tasks, including ontology classification, sentiment analysis, answer sentence
selection, and paraphrase identification, our proposed model, a modified CNN
with context-sensitive filters, consistently outperforms the standard CNN and
attention-based CNN baselines. By visualizing the learned context-sensitive
filters, we further validate and rationalize the effectiveness of proposed
framework.

本研究中提出一种使用小型元网络学习文本处理的上下文敏感卷积滤波器的方法，并将该框架进一步推广到模拟句子对，通过可视化学习的上下文敏感滤波器验证了所提出的框架的有效性。

学习上下文敏感的卷积滤波器 用于文本处理

Learning Context-Sensitive Convolutional Filters for Text Processing

Tree-structured neural networks have proven to be effective in learning
semantic representations by exploiting syntactic information. In spite of their
success, most existing models suffer from the underfitting problem: they
recursively use the same shared compositional function throughout the whole
compositional process and lack expressive power due to inability to capture the
richness of compositionality. In this paper, we address this issue by
introducing the dynamic compositional neural networks over tree structure
(DC-TreeNN), in which the compositional function is dynamically generated by a
meta network. The role of meta-network is to capture the metaknowledge across
the different compositional rules and formulate them. Experimental results on
two typical tasks show the effectiveness of the proposed models.

本文提出了基于动态组合神经网络的树状神经网络 (DC-TreeNN)，通过引入元网络动态生成组合函数，解决现有模型在组合过程中缺乏表达能力和容易欠拟合的问题，并在两个典型任务中取得了有效的实验结果。