Recently, reference-free metrics such as CLIPScore (Hessel et al., 2021) and
UMIC (Lee et al., 2021) have been proposed for automatic evaluation of image
captions, demonstrating a high correlation with human judgment. In this work,
our focus lies in evaluating the robustness of these metrics in scenarios that
require distinguishing between two captions with high lexical overlap but very
different meanings. Our findings reveal that despite their high correlation
with human judgment, both CLIPScore and UMIC struggle to identify fine-grained
errors in captions. However, when comparing different types of fine-grained
errors, both metrics exhibit limited sensitivity to implausibility of captions
and strong sensitivity to lack of sufficient visual grounding. Probing further
into the visual grounding aspect, we found that both CLIPScore and UMIC are
impacted by the size of image-relevant objects mentioned in the caption, and
that CLIPScore is also sensitive to the number of mentions of image-relevant
objects in the caption. In terms of linguistic aspects of a caption, we found
that both metrics lack the ability to comprehend negation, UMIC is sensitive to
caption lengths, and CLIPScore is insensitive to the structure of the sentence.
We hope our findings will serve as a valuable guide towards improving
reference-free evaluation in image captioning.

评估了 CLIPScore 和 UMIC 这两种自动评估图像标题的参考无关度量标准的鲁棒性，发现这两个指标在识别标题中的微小差错方面存在一定困难，同时受视觉概念和语言结构的影响。

无参考图像字幕评估指标的鲁棒性研究

An Examination of the Robustness of Reference-Free Image Captioning  Evaluation Metrics

Despite the success of various text generation metrics such as BERTScore, it
is still difficult to evaluate the image captions without enough reference
captions due to the diversity of the descriptions. In this paper, we introduce
a new metric UMIC, an Unreferenced Metric for Image Captioning which does not
require reference captions to evaluate image captions. Based on
Vision-and-Language BERT, we train UMIC to discriminate negative captions via
contrastive learning. Also, we observe critical problems of the previous
benchmark dataset (i.e., human annotations) on image captioning metric, and
introduce a new collection of human annotations on the generated captions. We
validate UMIC on four datasets, including our new dataset, and show that UMIC
has a higher correlation than all previous metrics that require multiple
references. We release the benchmark dataset and pre-trained models to compute
the UMIC.

本文提出了一种新的评估图像字幕的度量 UMIC，其通过对比学习不需要参考字幕。同时，我们发现以前的标准数据集在图像字幕度量方面存在关键问题，并对生成字幕进行了新的人类注释。在包括我们的新数据集在内的四个数据集上验证了 UMIC，证明了 UMIC 的相关性比所有需要多个参考文本度量的先前度量更高。我们发布了基准数据集和预训练模型以计算 UMIC。