Among the ever-evolving development of vision-language models, contrastive
language-image pretraining (CLIP) has set new benchmarks in many downstream
tasks such as zero-shot classifications by leveraging self-supervised
contrastive learning on large amounts of text-image pairs. However, its
dependency on rigid one-to-one mappings overlooks the complex and often
multifaceted relationships between and within texts and images. To this end, we
introduce RankCLIP, a novel pretraining method that extends beyond the rigid
one-to-one matching framework of CLIP and its variants. By leveraging both
in-modal and cross-modal ranking consistency, RankCLIP improves the alignment
process, enabling it to capture the nuanced many-to-many relationships between
and within each modality. Through comprehensive experiments, we demonstrate the
enhanced capability of RankCLIP to effectively improve performance across
various downstream tasks, notably achieving significant gains in zero-shot
classifications over state-of-the-art methods, underscoring the potential of
RankCLIP in further advancing vision-language pretraining.

通过在大量文本图像对上进行自我监督的对比学习，RankCLIP 在扩展 CLIP 的刚性一对一匹配框架的同时，利用模态内和跨模态的排序一致性来提高对齐过程，捕捉每种模态之间和内部的细致的多对多关系，有效提升各种下游任务的性能，尤其在零样本分类方面，显著超越现有方法，突显了 RankCLIP 在进一步推进视觉语言预训练方面的潜力。

RankCLIP: 语言 - 图像一致的排序预训练

RankCLIP: Ranking-Consistent Language-Image Pretraining

Emotion classification in text is a challenging and subjective task, due to
the involved cognitive inference processes that are required to interpret a
textual stimulus. In addition, the set of emotion categories is highly
domain-specific. For instance, literature analysis might require the use of
aesthetic emotions (e.g., finding something beautiful), and social media
analysis could benefit from fine-grained sets (e.g., separating anger from
annoyance) in contrast to basic emotion categories. This renders the task an
interesting field for zero-shot classifications, in which the label set is not
known at model development time. Unfortunately, most resources for emotion
analysis are English, and therefore, most studies on emotion analysis have been
performed in English, including those that involve prompting language models
for text labels. This leaves us with a research gap that we address in this
paper: In which language should we prompt for emotion labels on non-English
texts? This is particularly of interest when we have access to a multilingual
large language model, because we could request labels with English prompts even
for non-English data. Our experiments with natural language inference-based
language models show that it is consistently better to use English prompts even
if the data is in a different language.

情感分类是一个具有挑战性和主观性的任务，需要解释文本刺激所需的认知推理过程。本文针对存在的研究空白，探讨了在非英语文本中使用哪种语言进行情感标签提示的问题，并通过自然语言推理基于语言模型的实验结果表明，即使数据来源是不同语言，使用英语提示更为有效。