Recent progress in large-scale pre-training has led to the development of
advanced vision-language models (VLMs) with remarkable proficiency in
comprehending and generating multimodal content. Despite the impressive ability
to perform complex reasoning for VLMs, current models often struggle to
effectively and precisely capture the compositional information on both the
image and text sides. To address this, we propose FineMatch, a new aspect-based
fine-grained text and image matching benchmark, focusing on text and image
mismatch detection and correction. This benchmark introduces a novel task for
boosting and evaluating the VLMs' compositionality for aspect-based
fine-grained text and image matching. In this task, models are required to
identify mismatched aspect phrases within a caption, determine the aspect's
class, and propose corrections for an image-text pair that may contain between
0 and 3 mismatches. To evaluate the models' performance on this new task, we
propose a new evaluation metric named ITM-IoU for which our experiments show a
high correlation to human evaluation. In addition, we also provide a
comprehensive experimental analysis of existing mainstream VLMs, including
fully supervised learning and in-context learning settings. We have found that
models trained on FineMatch demonstrate enhanced proficiency in detecting
fine-grained text and image mismatches. Moreover, models (e.g., GPT-4V, Gemini
Pro Vision) with strong abilities to perform multimodal in-context learning are
not as skilled at fine-grained compositional image and text matching analysis.
With FineMatch, we are able to build a system for text-to-image generation
hallucination detection and correction.

提出了 FineMatch，一种新的面向细粒度文本和图像匹配的方面研究和评估 VLMs 组合性的基准，通过引入一个新的任务来检测和纠正文本和图像不匹配，对现有的主流 VLMs 进行了全面的实验分析。