Despite noise and caption quality having been acknowledged as important
factors impacting vision-language contrastive pre-training, in this paper, we
show that the full potential of improving the training process by addressing
such issues is yet to be realized. Specifically, we firstly study and analyze
two issues affecting training: incorrect assignment of negative pairs, and low
caption quality and diversity. Then, we devise effective solutions for
addressing both problems, which essentially require training with multiple true
positive pairs. Finally, we propose training with sigmoid loss to address such
a requirement. We show very large gains over the current state-of-the-art for
both image recognition ($\sim +6\%$ on average over 11 datasets) and image
retrieval ($\sim +19\%$ on Flickr30k and $\sim +15\%$ on MSCOCO).

本文研究视觉语言对比预训练中的问题，提出了解决负样本分配不正确和字幕质量低和多样性不足的有效方法，并通过使用 sigmoid loss 进行训练，在图像识别和图像检索方面取得了非常大的增益。

FFF: 修正有缺陷的基础对比预训练会得到非常强大的视觉 - 语言模型

FFF: Fixing Flawed Foundations in contrastive pre-training results in  very strong Vision-Language models

Contrastive learning has emerged as a prominent branch of self-supervised
learning for several years. Especially, CLIP, which applies contrastive
learning to large sets of captioned images, has garnered significant attention.
Recently, SigLIP, a variant of CLIP, has been proposed, which uses the sigmoid
loss instead of the standard InfoNCE loss. SigLIP achieves the performance
comparable to CLIP in a more efficient manner by eliminating the need for a
global view. However, theoretical understanding of using the sigmoid loss in
contrastive learning is underexplored. In this paper, we provide a theoretical
analysis of using the sigmoid loss in contrastive learning, in the perspective
of the geometric structure of learned embeddings. First, we propose the
double-Constant Embedding Model (CCEM), a framework for parameterizing various
well-known embedding structures by a single variable. Interestingly, the
proposed CCEM is proven to contain the optimal embedding with respect to the
sigmoid loss. Second, we mathematically analyze the optimal embedding
minimizing the sigmoid loss for contrastive learning. The optimal embedding
ranges from simplex equiangular-tight-frame to antipodal structure, depending
on the temperature parameter used in the sigmoid loss. Third, our experimental
results on synthetic datasets coincide with the theoretical results on the
optimal embedding structures.

对使用 sigmoid 损失进行对比学习的理论分析，使用双常数嵌入模型（CCEM）参数化多种已知嵌入结构，数学分析 sigmoid 损失下的最优嵌入，实验结果与理论结果一致。

使用 Sigmoid Loss 进行对比学习的分析

Analysis of Using Sigmoid Loss for Contrastive Learning

We propose a simple pairwise sigmoid loss for image-text pre-training. Unlike
standard contrastive learning with softmax normalization, the sigmoid loss
operates solely on image-text pairs and does not require a global view of the
pairwise similarities for normalization. The sigmoid loss simultaneously allows
further scaling up the batch size, while also performing better at smaller
batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at 4k
batch size and a Large LiT model at 20k batch size, the latter achieves 84.5%
ImageNet zero-shot accuracy in two days. This disentanglement of the batch size
from the loss further allows us to study the impact of examples vs pairs and
negative to positive ratio. Finally, we push the batch size to the extreme, up
to one million, and find that the benefits of growing batch size quickly
diminish, with a more reasonable batch size of 32k being sufficient. We hope
our research motivates further explorations in improving the quality and
efficiency of language-image pre-training.

本文提出了针对图像 - 文本预训练的简单对数损失函数，其中使用 sigmoid 函数，从而可以放大批量大小，并实现更好的性能表现。