Compositional Reasoning (CR) entails grasping the significance of attributes,
relations, and word order. Recent Vision-Language Models (VLMs), comprising a
visual encoder and a Large Language Model (LLM) decoder, have demonstrated
remarkable proficiency in such reasoning tasks. This prompts a crucial
question: have VLMs effectively tackled the CR challenge? We conjecture that
existing CR benchmarks may not adequately push the boundaries of modern VLMs
due to the reliance on an LLM-only negative text generation pipeline.
Consequently, the negatives produced either appear as outliers from the natural
language distribution learned by VLMs' LLM decoders or as improbable within the
corresponding image context. To address these limitations, we introduce ConMe
-- a compositional reasoning benchmark and a novel data generation pipeline
leveraging VLMs to produce `hard CR Q&A'. Through a new concept of VLMs
conversing with each other to collaboratively expose their weaknesses, our
pipeline autonomously generates, evaluates, and selects challenging
compositional reasoning questions, establishing a robust CR benchmark, also
subsequently validated manually. Our benchmark provokes a noteworthy, up to
33%, decrease in CR performance compared to preceding benchmarks, reinstating
the CR challenge even for state-of-the-art VLMs.

我们提出了一种新的数据生成流水线 ConMe，利用视觉 - 语言模型 (VLMs) 产生具有挑战性的复合推理问题，并通过与同类模型协同对话揭示其弱点，建立了一个稳健的复合推理基准，验证表明我们的基准相对于以前的基准提高了高达 33% 的复合推理性能。