Knowledge distillation becomes a de facto standard to improve the performance
of small neural networks. Most of the previous works propose to regress the
representational features from the teacher to the student in a one-to-one
spatial matching fashion. However, people tend to overlook the fact that, due
to the architecture differences, the semantic information on the same spatial
location usually vary. This greatly undermines the underlying assumption of the
one-to-one distillation approach. To this end, we propose a novel one-to-all
spatial matching knowledge distillation approach. Specifically, we allow each
pixel of the teacher feature to be distilled to all spatial locations of the
student features given its similarity, which is generated from a target-aware
transformer. Our approach surpasses the state-of-the-art methods by a
significant margin on various computer vision benchmarks, such as ImageNet,
Pascal VOC and COCOStuff10k. Code will be released soon.

本篇论文提出了一种新型的知识蒸馏方法，采用一对所有的空间匹配，提高小型神经网络的性能，不同于以往的一对一的空间匹配，从而导致所有的空间位置通常都具有不同的语义信息。该方法在各种计算机视觉基准测试中都超过了最先进的方法。

目标感知 Transformer 的知识蒸馏

Knowledge Distillation via the Target-aware Transformer

Visual place recognition is a challenging task for applications such as
autonomous driving navigation and mobile robot localization. Distracting
elements presenting in complex scenes often lead to deviations in the
perception of visual place. To address this problem, it is crucial to integrate
information from only task-relevant regions into image representations. In this
paper, we introduce a novel holistic place recognition model, TransVPR, based
on vision Transformers. It benefits from the desirable property of the
self-attention operation in Transformers which can naturally aggregate
task-relevant features. Attentions from multiple levels of the Transformer,
which focus on different regions of interest, are further combined to generate
a global image representation. In addition, the output tokens from Transformer
layers filtered by the fused attention mask are considered as key-patch
descriptors, which are used to perform spatial matching to re-rank the
candidates retrieved by the global image features. The whole model allows
end-to-end training with a single objective and image-level supervision.
TransVPR achieves state-of-the-art performance on several real-world benchmarks
while maintaining low computational time and storage requirements.

本文介绍了一种新颖的基于 Transformer 的全局位置识别模型 TransVPR，该模型在多尺度上聚合任务相关特征，并通过空间匹配实现对全局视觉特征的候选人重新排序，具有最先进的性能并且计算时间和存储要求相对较低。