We propose CounterCurate, a framework to comprehensively improve the
visio-linguistic compositional reasoning capability for both contrastive and
generative multimodal models. In particular, we identify two under-explored
critical problems: the neglect of the physically grounded reasoning (counting
and position understanding) and the potential of using highly capable text and
image generation models for semantic counterfactual fine-tuning. Our work
pioneers an approach that addresses these gaps. We first spotlight the
near-chance performance of multimodal models like CLIP and LLaVA in physically
grounded compositional reasoning. We then apply simple data augmentation using
a grounded image generation model, GLIGEN, to generate finetuning data,
resulting in significant performance improvements: +33% and +37% for CLIP and
LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark.
Moreover, we exploit the capabilities of high-performing text generation and
image generation models, specifically GPT-4V and DALLE-3, to curate challenging
semantic counterfactuals, thereby further enhancing compositional reasoning
capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms
GPT-4V.

通过 CounterCurate 框架，综合改善对比式和生成式多模态模型的视觉 - 语言组合推理能力，通过解决忽视物理基础推理（计数和位置理解）和使用高能力文本和图像生成模型进行语义反事实微调的潜力等两个问题来提高推理性能。

CounterCurate：通过反事实的例子增强物理和语义的视觉 - 语言组合推理能力

CounterCurate: Enhancing Physical and Semantic Visio-Linguistic  Compositional Reasoning via Counterfactual Examples

We present a novel task and dataset for evaluating the ability of vision and
language models to conduct visio-linguistic compositional reasoning, which we
call Winoground. Given two images and two captions, the goal is to match them
correctly - but crucially, both captions contain a completely identical set of
words, only in a different order. The dataset was carefully hand-curated by
expert annotators and is labeled with a rich set of fine-grained tags to assist
in analyzing model performance. We probe a diverse range of state-of-the-art
vision and language models and find that, surprisingly, none of them do much
better than chance. Evidently, these models are not as skilled at
visio-linguistic compositional reasoning as we might have hoped. We perform an
extensive analysis to obtain insights into how future work might try to
mitigate these models' shortcomings. We aim for Winoground to serve as a useful
evaluation set for advancing the state of the art and driving further progress
in the field. The dataset is available at
this https URL.

介绍了一项新的任务和数据集 Winoground，用于评估视觉和语言模型进行视觉 - 语言组成推理的能力，展示了目前最先进的视觉和语言模型在该任务上表现不尽如人意的情况，并探讨如何改进模型的方法。