Soft targets combined with the cross-entropy loss have shown to improve
generalization performance of deep neural networks on supervised classification
tasks. The standard cross-entropy loss however assumes data to be categorically
distributed, which may often not be the case in practice. In contrast, InfoNCE
does not rely on such an explicit assumption but instead implicitly estimates
the true conditional through negative sampling. Unfortunately, it cannot be
combined with soft targets in its standard formulation, hindering its use in
combination with sophisticated training strategies. In this paper, we address
this limitation by proposing a principled loss function that is compatible with
probabilistic targets. Our new soft target InfoNCE loss is conceptually simple,
efficient to compute, and can be derived within the framework of noise
contrastive estimation. Using a toy example, we demonstrate shortcomings of the
categorical distribution assumption of cross-entropy, and discuss implications
of sampling from soft distributions. We observe that soft target InfoNCE
performs on par with strong soft target cross-entropy baselines and outperforms
hard target NLL and InfoNCE losses on popular benchmarks, including ImageNet.
Finally, we provide a simple implementation of our loss, geared towards
supervised classification and fully compatible with deep classification model
trained with cross-entropy.

通过提出一个新的损失函数，本文解决了 InfoNCE 无法与软目标结合使用的限制，并在常见基准测试中表现优异，包括 ImageNet。

使用软目标进行条件模型的噪声对比估计

Noise contrastive estimation with soft targets for conditional models

Plant leaf identification is crucial for biodiversity protection and
conservation and has gradually attracted the attention of academia in recent
years. Due to the high similarity among different varieties, leaf cultivar
recognition is also considered to be an ultra-fine-grained visual
classification (UFGVC) task, which is facing a huge challenge. In practice, an
instance may be related to multiple varieties to varying degrees, especially in
the UFGVC datasets. However, deep learning methods trained on one-hot labels
fail to reflect patterns shared across categories and thus perform poorly on
this task. To address this issue, we generate soft targets integrated with
inter-class similarity information. Specifically, we continuously update the
prototypical features for each category and then capture the similarity scores
between instances and prototypes accordingly. Original one-hot labels and the
similarity scores are incorporated to yield enhanced labels. Prototype-enhanced
soft labels not only contain original one-hot label information, but also
introduce rich inter-category semantic association information, thus providing
more effective supervision for deep model training. Extensive experimental
results on public datasets show that our method can significantly improve the
performance on the UFGVC task of leaf cultivar identification.

本研究提出了一种深度学习方法，通过生成软目标来提供更有效的监督，并结合原始的 one-hot 标签信息和实例与原型之间的相似度评分，以在植物叶子识别中实现更好的性能。

基于原型增强学习的叶片品种识别

Leaf Cultivar Identification via Prototype-enhanced Learning

This article sets forth a review of knowledge distillation techniques with a
focus on their applicability to retail banking contexts. Predictive machine
learning algorithms used in banking environments, especially in risk and
control functions, are generally subject to regulatory and technical
constraints limiting their complexity. Knowledge distillation gives the
opportunity to improve the performances of simple models without burdening
their application, using the results of other - generally more complex and
better-performing - models. Parsing recent advances in this field, we highlight
three main approaches: Soft Targets, Sample Selection and Data Augmentation. We
assess the relevance of a subset of such techniques by applying them to open
source datasets, before putting them to the test on the use cases of BPCE, a
major French institution in the retail banking sector. As such, we demonstrate
the potential of knowledge distillation to improve the performance of these
models without altering their form and simplicity.

本研究探讨知识蒸馏技术在零售银行情境中的适用性，其中使用软目标、样本选择和数据增强三种方法来提高简化模型性能，对 BPCE 进行实证测试并证明知识蒸馏可以在不改变模型形式和简单性的情况下，提高其性能。

使用知识蒸馏技术在零售银行场景中优化可解释模型

Using Knowledge Distillation to improve interpretable models in a retail banking context

There is growing evidence that converting targets to soft targets in
supervised learning can provide considerable gains in performance. Much of this
work has considered classification, converting hard zero-one values to soft
labels---such as by adding label noise, incorporating label ambiguity or using
distillation. In parallel, there is some evidence from a regression setting in
reinforcement learning that learning distributions can improve performance. In
this work, we investigate the reasons for this improvement, in a regression
setting. We introduce a novel distributional regression loss, and similarly
find it significantly improves prediction accuracy. We investigate several
common hypotheses, around reducing overfitting and improved representations. We
instead find evidence for an alternative hypothesis: this loss is easier to
optimize, with better behaved gradients, resulting in improved generalization.
We provide theoretical support for this alternative hypothesis, by
characterizing the norm of the gradients of this loss.

在多个学科领域中发现，将目标转换为软目标可提高性能，本研究考察了分布式回归损失（distributional regression loss）在学习分布时的表现，并发现它在改善预测准确率方面显著提高，我们提供了理论支持，表明此损失由于其更好的梯度表现，易于优化导致了这种改进。