The Balanced-Pairwise-Affinities (BPA) feature transform is designed to
upgrade the features of a set of input items to facilitate downstream matching
or grouping related tasks. The transformed set encodes a rich representation of
high order relations between the input features. A particular min-cost-max-flow
fractional matching problem, whose entropy regularized version can be
approximated by an optimal transport (OT) optimization, leads to a transform
which is efficient, differentiable, equivariant, parameterless and
probabilistically interpretable. While the Sinkhorn OT solver has been adapted
extensively in many contexts, we use it differently by minimizing the cost
between a set of features to $itself$ and using the transport plan's $rows$ as
the new representation. Empirically, the transform is highly effective and
flexible in its use and consistently improves networks it is inserted into, in
a variety of tasks and training schemes. We demonstrate state-of-the-art
results in few-shot classification, unsupervised image clustering and person
re-identification. Code is available at \url{github.com/DanielShalam/BPA}.

设计了平衡成对亲和力（BPA）特征转换以升级一组输入项的特征，使其适用于下游匹配或相关任务。通过最小成本最大流分数匹配问题、基于最优传输（OT）优化来近似其熵正则化版本，得到一种高效、可微分、等变、无参数且具有概率解释性的转换。它在各种任务和训练方案中都表现出极高的效果和灵活性，并且在少样本分类、无监督图像聚类和人物重新识别等方面展示出最先进的结果。

平衡对比特征变换

The Balanced-Pairwise-Affinities Feature Transform

Typical technique in knowledge distillation (KD) is regularizing the learning
of a limited capacity model (student) by pushing its responses to match a
powerful model's (teacher). Albeit useful especially in the penultimate layer
and beyond, its action on student's feature transform is rather implicit,
limiting its practice in the intermediate layers. To explicitly embed the
teacher's knowledge in feature transform, we propose a learnable KD layer for
the student which improves KD with two distinct abilities: i) learning how to
leverage the teacher's knowledge, enabling to discard nuisance information, and
ii) feeding forward the transferred knowledge deeper. Thus, the student enjoys
the teacher's knowledge during the inference besides training. Formally, we
repurpose 1x1-BN-ReLU-1x1 convolution block to assign a semantic vector to each
local region according to the template (supervised by the teacher) that the
corresponding region of the student matches. To facilitate template learning in
the intermediate layers, we propose a novel form of supervision based on the
teacher's decisions. Through rigorous experimentation, we demonstrate the
effectiveness of our approach on 3 popular classification benchmarks. Code is
available at: this https URL

通过引入可学习的 KD 层和模板学习方法，我们提出一种新的知识蒸馏技术，实现了对学生模型在中间层中进行特征变换的显式控制，并在多个分类基准测试中验证了其有效性。

让学生决策的知识蒸馏层

Knowledge Distillation Layer that Lets the Student Decide

Existing feature distillation methods commonly adopt the One-to-one
Representation Matching between any pre-selected teacher-student layer pair. In
this paper, we present N-to-One Representation (NORM), a new two-stage
knowledge distillation method, which relies on a simple Feature Transform (FT)
module consisting of two linear layers. In view of preserving the intact
information learnt by the teacher network, during training, our FT module is
merely inserted after the last convolutional layer of the student network. The
first linear layer projects the student representation to a feature space
having N times feature channels than the teacher representation from the last
convolutional layer, and the second linear layer contracts the expanded output
back to the original feature space. By sequentially splitting the expanded
student representation into N non-overlapping feature segments having the same
number of feature channels as the teacher's, they can be readily forced to
approximate the intact teacher representation simultaneously, formulating a
novel many-to-one representation matching mechanism conditioned on a single
teacher-student layer pair. After training, such an FT module will be naturally
merged into the subsequent fully connected layer thanks to its linear property,
introducing no extra parameters or architectural modifications to the student
network at inference. Extensive experiments on different visual recognition
benchmarks demonstrate the leading performance of our method. For instance, the
ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches
72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a
pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an
absolute improvement of 2.01%|4.63%|3.03% against the individually trained
counterpart. Code is available at this https URL

本研究提出了 N-to-One Representation Matching (NORM) 知识蒸馏方法，其主要利用由两个线性层组成的 Feature Transform (FT) 模块，可以在训练中保留教师网络学习到的信息，并且通过新颖的多对一表示匹配机制，推动学生网络逼近教师网络的表示。 在不引入额外参数或结构修改的情况下，在多个视觉识别基准测试中展示了该方法的领先性能。

NORM: 基于 N 对一表示匹配的知识蒸馏

NORM: Knowledge Distillation via N-to-One Representation Matching

Universal style transfer tries to explicitly minimize the losses in feature
space, thus it does not require training on any pre-defined styles. It usually
uses different layers of VGG network as the encoders and trains several
decoders to invert the features into images. Therefore, the effect of style
transfer is achieved by feature transform. Although plenty of methods have been
proposed, a theoretical analysis of feature transform is still missing. In this
paper, we first propose a novel interpretation by treating it as the optimal
transport problem. Then, we demonstrate the relations of our formulation with
former works like Adaptive Instance Normalization (AdaIN) and Whitening and
Coloring Transform (WCT). Finally, we derive a closed-form solution named
Optimal Style Transfer (OST) under our formulation by additionally considering
the content loss of Gatys. Comparatively, our solution can preserve better
structure and achieve visually pleasing results. It is simple yet effective and
we demonstrate its advantages both quantitatively and qualitatively. Besides,
we hope our theoretical analysis can inspire future works in neural style
transfer. Code is available at this https URL

通过将其视为最优输运问题（Optimal Transport Problem），我们提出了一种新颖的样式转移解释方法，并证明了与 AdaIN 和 WCT 之类的旧方法的关系，并在考虑了 Gatys 的内容损失后得出了闭合形式的最优样式转移（OST）解决方案。