Existing vision-language models (VLMs) treat text descriptions as a unit,
confusing individual concepts in a prompt and impairing visual semantic
matching and reasoning. An important aspect of reasoning in logic and language
is negations. This paper highlights the limitations of popular VLMs such as
CLIP, at understanding the implications of negations, i.e., the effect of the
word "not" in a given prompt. To enable evaluation of VLMs on fluent prompts
with negations, we present CC-Neg, a dataset containing 228,246 images, true
captions and their corresponding negated captions. Using CC-Neg along with
modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework,
has an improved understanding of negations. This training paradigm improves
CoN-CLIP's ability to encode semantics reliably, resulting in 3.85% average
gain in top-1 accuracy for zero-shot image classification across 8 datasets.
Further, CoN-CLIP outperforms CLIP on challenging compositionality benchmarks
such as SugarCREPE by 4.4%, showcasing emergent compositional understanding of
objects, relations, and attributes in text. Overall, our work addresses a
crucial limitation of VLMs by introducing a dataset and framework that
strengthens semantic associations between images and text, demonstrating
improved large-scale foundation models with significantly reduced computational
cost, promoting efficiency and accessibility.

通过引入 CC-Neg 数据集和 CoN-CLIP 框架，我们的工作解决了视觉语言模型中一个重要的局限性，加强了图像和文本之间的语义关联，展示了具有显著降低计算成本的改进型大规模基础模型，提升了效率和可访问性。