Referenceless metrics (e.g., CLIPScore) use pretrained vision--language
models to assess image descriptions directly without costly ground-truth
reference texts. Such methods can facilitate rapid progress, but only if they
truly align with human preference judgments. In this paper, we introduce
ContextRef, a benchmark for assessing referenceless metrics for such alignment.
ContextRef has two components: human ratings along a variety of established
quality dimensions, and ten diverse robustness checks designed to uncover
fundamental weaknesses. A crucial aspect of ContextRef is that images and
descriptions are presented in context, reflecting prior work showing that
context is important for description quality. Using ContextRef, we assess a
variety of pretrained models, scoring functions, and techniques for
incorporating context. None of the methods is successful with ContextRef, but
we show that careful fine-tuning yields substantial improvements. ContextRef
remains a challenging benchmark though, in large part due to the challenge of
context dependence.

该研究引入了 ContextRef 作为一个基准，用于评估视觉 - 语言模型的参考度量与人类喜好判断的一致性，并通过评估各种预训练模型、评分函数和上下文整合技术来提高描述质量。

图像描述生成的无参考度量评估

ContextRef: Evaluating Referenceless Metrics For Image Description  Generation

Few images on the Web receive alt-text descriptions that would make them
accessible to blind and low vision (BLV) users. Image-based NLG systems have
progressed to the point where they can begin to address this persistent
societal problem, but these systems will not be fully successful unless we
evaluate them on metrics that guide their development correctly. Here, we argue
against current referenceless metrics -- those that don't rely on
human-generated ground-truth descriptions -- on the grounds that they do not
align with the needs of BLV users. The fundamental shortcoming of these metrics
is that they do not take context into account, whereas contextual information
is highly valued by BLV users. To substantiate these claims, we present a study
with BLV participants who rated descriptions along a variety of dimensions. An
in-depth analysis reveals that the lack of context-awareness makes current
referenceless metrics inadequate for advancing image accessibility. As a
proof-of-concept, we provide a contextual version of the referenceless metric
CLIPScore which begins to address the disconnect to the BLV data. An accessible
HTML version of this paper is available at
this https URL

在对盲人和低视力用户进行调查后，本研究认为目前的基于图像的自然语言生成系统无法满足用户的需求，因为现有的评价指标没有考虑上下文信息，而上下文信息对于盲人和低视力用户至关重要。为解决这个问题，本文提出一个新的基于上下文信息的评价指标，并对其进行了证明。