Contrastive language-image pre-training (CLIP) models have demonstrated
considerable success across various vision-language tasks, such as
text-to-image retrieval, where the model is required to effectively process
natural language input to produce an accurate visual output. However, current
models still face limitations in dealing with linguistic variations in input
queries, such as paraphrases, making it challenging to handle a broad range of
user queries in real-world applications. In this study, we introduce a
straightforward fine-tuning approach to enhance the representations of CLIP
models for paraphrases. Our approach involves a two-step paraphrase generation
process, where we automatically create two categories of paraphrases from
web-scale image captions by leveraging large language models. Subsequently, we
fine-tune the CLIP text encoder using these generated paraphrases while
freezing the image encoder. Our resulting model, which we call ParaCLIP,
exhibits significant improvements over baseline CLIP models across various
tasks, including paraphrased retrieval (with rank similarity scores improved by
up to 2.0% and 5.6%), Visual Genome Relation and Attribution, as well as seven
semantic textual similarity tasks.

通过引入一个简单的微调方法，我们提出了一种改进 CLIP 模型对于释义的表征的方法，该方法使用大规模语言模型从 Web 规模的图像字幕中自动创建两类释义，并在固定图像编码器的同时微调 CLIP 文本编码器。我们的模型 ParaCLIP 在各种任务中都显示出与基准 CLIP 模型相比的显著改进，包括释义检索、Visual Genome 关系和属性，以及七个语义文本相似度任务。

用两步重述细调 CLIP 文本编码器

Fine-tuning CLIP Text Encoders with Two-step Paraphrasing

We present Unsupervised hard Negative Augmentation (UNA), a method that
generates synthetic negative instances based on the term frequency-inverse
document frequency (TF-IDF) retrieval model. UNA uses TF-IDF scores to
ascertain the perceived importance of terms in a sentence and then produces
negative samples by replacing terms with respect to that. Our experiments
demonstrate that models trained with UNA improve the overall performance in
semantic textual similarity tasks. Additional performance gains are obtained
when combining UNA with the paraphrasing augmentation. Further results show
that our method is compatible with different backbone models. Ablation studies
also support the choice of having a TF-IDF-driven control on negative
augmentation.

我们提出了一种无监督的困难负面增强 (UNA) 方法，该方法基于词频 - 逆文档频率 (TF-IDF) 检索模型生成合成的负面实例。 UNA 使用 TF-IDF 分数确定句子中术语的感知重要性，然后通过替换术语来生成负面样本。我们的实验证明，使用 UNA 转换的模型在语义文本相似性任务中改善了整体性能。与重述增强相结合后，还可以获得额外的性能提升。进一步的结果表明，我们的方法与不同的骨干模型兼容。消融研究还支持在负面增强中选择以 TF-IDF 为驱动的控制。

无监督难负样本增强的对比学习

Unsupervised hard Negative Augmentation for contrastive learning

Prior studies diagnose the anisotropy problem in sentence representations
from pre-trained language models, e.g., BERT, without fine-tuning. Our analysis
reveals that the sentence embeddings from BERT suffer from a bias towards
uninformative words, limiting the performance in semantic textual similarity
(STS) tasks. To address this bias, we propose a simple and efficient
unsupervised approach, Diagonal Attention Pooling (Ditto), which weights words
with model-based importance estimations and computes the weighted average of
word representations from pre-trained models as sentence embeddings. Ditto can
be easily applied to any pre-trained language model as a postprocessing
operation. Compared to prior sentence embedding approaches, Ditto does not add
parameters nor requires any learning. Empirical evaluations demonstrate that
our proposed Ditto can alleviate the anisotropy problem and improve various
pre-trained models on STS tasks.

该研究提出了一种名为 Diagonal Attention Pooling (Ditto) 的无监督方法来解决 BERT 等预训练语言模型中句子表示的各向异性问题，通过基于模型的重要性估计对单词进行加权，改进了语义文本相似度 (STS) 任务的性能。