Recent progress in large-scale pre-training has led to the development of
advanced vision-language models (VLMs) with remarkable proficiency in
comprehending and generating multimodal content. Despite the impressive ability
to perform complex reasoning for VLMs, current models often struggle to
effectively and precisely capture the compositional information on both the
image and text sides. To address this, we propose FineMatch, a new aspect-based
fine-grained text and image matching benchmark, focusing on text and image
mismatch detection and correction. This benchmark introduces a novel task for
boosting and evaluating the VLMs' compositionality for aspect-based
fine-grained text and image matching. In this task, models are required to
identify mismatched aspect phrases within a caption, determine the aspect's
class, and propose corrections for an image-text pair that may contain between
0 and 3 mismatches. To evaluate the models' performance on this new task, we
propose a new evaluation metric named ITM-IoU for which our experiments show a
high correlation to human evaluation. In addition, we also provide a
comprehensive experimental analysis of existing mainstream VLMs, including
fully supervised learning and in-context learning settings. We have found that
models trained on FineMatch demonstrate enhanced proficiency in detecting
fine-grained text and image mismatches. Moreover, models (e.g., GPT-4V, Gemini
Pro Vision) with strong abilities to perform multimodal in-context learning are
not as skilled at fine-grained compositional image and text matching analysis.
With FineMatch, we are able to build a system for text-to-image generation
hallucination detection and correction.

提出了 FineMatch，一种新的面向细粒度文本和图像匹配的方面研究和评估 VLMs 组合性的基准，通过引入一个新的任务来检测和纠正文本和图像不匹配，对现有的主流 VLMs 进行了全面的实验分析。

FINEMATCH：基于方面的细粒度图像与文本不匹配检测与校正

FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection  and Correction

Probabilistic models analyze data by relying on a set of assumptions. Data
that exhibit deviations from these assumptions can undermine inference and
prediction quality. Robust models offer protection against mismatch between a
model's assumptions and reality. We propose a way to systematically detect and
mitigate mismatch of a large class of probabilistic models. The idea is to
raise the likelihood of each observation to a weight and then to infer both the
latent variables and the weights from data. Inferring the weights allows a
model to identify observations that match its assumptions and down-weight
others. This enables robust inference and improves predictive accuracy. We
study four different forms of mismatch with reality, ranging from missing
latent groups to structure misspecification. A Poisson factorization analysis
of the Movielens 1M dataset shows the benefits of this approach in a practical
scenario.

本研究提出了一种方法，通过将每个观察的可能性变为一种权重，从数据中推断连续变量和权重，以检测和减轻大量概率模型的不匹配性，并明确不同类型的不匹配性对模型的影响与表现，这有助于提高模型的鲁棒性和预测准确性。