Large-scale pre-trained Vision & Language (VL) models have shown remarkable
performance in many applications, enabling replacing a fixed set of supported
classes with zero-shot open vocabulary reasoning over (almost arbitrary)
natural language prompts. However, recent works have uncovered a fundamental
weakness of these models. For example, their difficulty to understand Visual
Language Concepts (VLC) that go 'beyond nouns' such as the meaning of
non-object words (e.g., attributes, actions, relations, states, etc.), or
difficulty in performing compositional reasoning such as understanding the
significance of the order of the words in a sentence. In this work, we
investigate to which extent purely synthetic data could be leveraged to teach
these models to overcome such shortcomings without compromising their zero-shot
capabilities. We contribute Synthetic Visual Concepts (SyViC) - a million-scale
synthetic dataset and data generation codebase allowing to generate additional
suitable data to improve VLC understanding and compositional reasoning of VL
models. Additionally, we propose a general VL finetuning strategy for
effectively leveraging SyViC towards achieving these improvements. Our
extensive experiments and ablations on VL-Checklist, Winoground, and ARO
benchmarks demonstrate that it is possible to adapt strong pre-trained VL
models with synthetic data significantly enhancing their VLC understanding
(e.g. by 9.9% on ARO and 4.3% on VL-Checklist) with under 1% drop in their
zero-shot accuracy.

本文旨在通过提供一百万级的、人工合成的图像数据集 SyViC 和相应的数据生成代码，以及细调模型的策略，探讨如何提高视觉语言模型对非物体类词汇等方面（即超越名词的视觉语言概念）的理解和组合推理能力，从而在保持零样本准确度的前提下，极大提高模型的性能。通过在 ARO 和 VL-Checklist 等基准测试上的广泛实验和削减，我们证明了用人工合成数据进行训练可以在不牺牲零样本能力的情况下大幅提升其 VLC 理解能力（如在 ARO 上提高 9.9％，VL-Checklist 上提高 4.3％）。