This paper presents an analysis regarding an influence of the Distance Metric
Learning (DML) loss functions on the supervised fine-tuning of the language
models for classification tasks. We experimented with known datasets from
SentEval Transfer Tasks.
Our experiments show that applying the DML loss function can increase
performance on downstream classification tasks of RoBERTa-large models in
few-shot scenarios. Models fine-tuned with the use of SoftTriple loss can
achieve better results than models with a standard categorical cross-entropy
loss function by about 2.89 percentage points from 0.04 to 13.48 percentage
points depending on the training dataset. Additionally, we accomplished a
comprehensive analysis with explainability techniques to assess the models'
reliability and explain their results.

本文研究了距离度量学习 (Distance Metric Learning，DML) 损失函数对用于分类任务的语言模型有监督微调的影响。在实验中，我们使用 SentEval Transfer Tasks 数据集，发现使用 DML 损失函数可以在少样本情况下提高 RoBERTa-large 模型在下游分类任务中的性能，使用 SoftTriple 损失微调的模型可以比使用标准分类交叉熵损失函数的模型取得更好的结果。同时，我们使用可解释性技术对模型的可靠性和结果进行了全面分析。

监督式语言模型微调的小样本场景距离度量损失函数

Distance Metric Learning Loss Functions in Few-Shot Scenarios of Supervised Language Models Fine-Tuning

Distance metric learning (DML) is to learn the embeddings where examples from
the same class are closer than examples from different classes. It can be cast
as an optimization problem with triplet constraints. Due to the vast number of
triplet constraints, a sampling strategy is essential for DML. With the
tremendous success of deep learning in classifications, it has been applied for
DML. When learning embeddings with deep neural networks (DNNs), only a
mini-batch of data is available at each iteration. The set of triplet
constraints has to be sampled within the mini-batch. Since a mini-batch cannot
capture the neighbors in the original set well, it makes the learned embeddings
sub-optimal. On the contrary, optimizing SoftMax loss, which is a
classification loss, with DNN shows a superior performance in certain DML
tasks. It inspires us to investigate the formulation of SoftMax. Our analysis
shows that SoftMax loss is equivalent to a smoothed triplet loss where each
class has a single center. In real-world data, one class can contain several
local clusters rather than a single one, e.g., birds of different poses.
Therefore, we propose the SoftTriple loss to extend the SoftMax loss with
multiple centers for each class. Compared with conventional deep metric
learning algorithms, optimizing SoftTriple loss can learn the embeddings
without the sampling phase by mildly increasing the size of the last fully
connected layer. Experiments on the benchmark fine-grained data sets
demonstrate the effectiveness of the proposed loss function. Code is available
at this https URL

通过使用 SoftTriple loss 函数中每一个类别拥有多个中心的方式，可优化浅层距离计量学习算法，而无需采样。