Evaluating the compatibility between textual descriptions and corresponding
images represents a core endeavor within multi-modal research. In recent years,
a proliferation of reference-free methods, leveraging visual-language
pre-trained models (VLMs), has emerged. Empirical evidence has substantiated
that these innovative approaches exhibit a higher correlation with human
judgment, marking a significant advancement in the field. However, does a
higher correlation with human evaluations alone sufficiently denote the
complete of a metric? In response to this question, in this paper, we study if
there are any deficiencies in reference-free metrics. Specifically, inspired by
the Cobra Effect, we utilize metric scores as rewards to direct the captioning
model toward generating descriptions that closely align with the metric's
criteria. If a certain metric has flaws, it will be exploited by the model and
reflected in the generated sentences. Our findings reveal that descriptions
guided by these metrics contain significant flaws, e.g. incoherent statements
and excessive repetition. Subsequently, we propose a novel method termed
Self-Improving to rectify the identified shortcomings within these metrics. We
employ GPT-4V as an evaluative tool to assess generated sentences and the
result reveals that our approach achieves state-of-the-art (SOTA) performance.
In addition, we also introduce a challenging evaluation benchmark called Flaws
Caption to evaluate reference-free image captioning metrics comprehensively.
Our code is available at
this https URL

评估文本描述与相应图像之间的兼容性是多模态研究中的核心工作之一。本文研究了无参考指标的不足之处，并提出了一种名为 “自我完善” 的新方法来纠正这些指标的不足，并通过 GPT-4V 评估生成的句子以获得最先进的性能。此外，我们还介绍了一个具有挑战性的评估基准来全面评估无参考图像描述度量标准。