In this paper, we introduce a novel approach to novel object captioning which
employs relative contrastive learning to learn visual and semantic alignment.
Our approach maximizes compatibility between regions and object tags in a
contrastive manner. To set up a proper contrastive learning objective, for each
image, we augment tags by leveraging the relative nature of positive and
negative pairs obtained from foundation models such as CLIP. We then use the
rank of each augmented tag in a list as a relative relevance label to contrast
each top-ranked tag with a set of lower-ranked tags. This learning objective
encourages the top-ranked tags to be more compatible with their image and text
context than lower-ranked tags, thus improving the discriminative ability of
the learned multi-modality representation. We evaluate our approach on two
datasets and show that our proposed RCA-NOC approach outperforms
state-of-the-art methods by a large margin, demonstrating its effectiveness in
improving vision-language representation for novel object captioning.

通过相对对比学习，本研究提出了一种新颖的方法来学习视觉和语义对齐，从而实现新颖物体的描述。针对每个图像，通过利用基于 CLIP 的正负样本的相对性质增加标签，设置适当的对比学习目标，并且将每个增强标签在列表中的排名作为相对相关性标签来对比每个排名靠前的标签和一组排名较低的标签。通过这个学习目标，使得排名靠前的标签与图像和文本上下文的兼容性比排名较低的标签更好，从而提高了学习到的多模态表示的判别能力。在两个数据集上对这种方法进行评估，并且显示了 RCA-NOC 方法在改进新颖物体描述的视觉语言表示方面的显著优势，证明了其有效性。