Text-to-image (T2I) synthesis has recently achieved significant advancements.
However, challenges remain in the model's compositionality, which is the
ability to create new combinations from known components. We introduce
Winoground-T2I, a benchmark designed to evaluate the compositionality of T2I
models. This benchmark includes 11K complex, high-quality contrastive sentence
pairs spanning 20 categories. These contrastive sentence pairs with subtle
differences enable fine-grained evaluations of T2I synthesis models.
Additionally, to address the inconsistency across different metrics, we propose
a strategy that evaluates the reliability of various metrics by using
comparative sentence pairs. We use Winoground-T2I with a dual objective: to
evaluate the performance of T2I models and the metrics used for their
evaluation. Finally, we provide insights into the strengths and weaknesses of
these metrics and the capabilities of current T2I models in tackling challenges
across a range of complex compositional categories. Our benchmark is publicly
available at this https URL .

Winoground-T2I 是一个用于评估 T2I 模型组成性的基准，通过包含 11K 个复杂、高质量的对比句对来进行评估，通过比较性句对来评估各种指标的可靠性，最后提供了关于指标的优点和缺点以及当前 T2I 模型在应对复杂组成类别挑战中的能力，该基准公开提供。