The evaluation of text-generative vision-language models is a challenging yet
crucial endeavor. By addressing the limitations of existing Visual Question
Answering (VQA) benchmarks and proposing innovative evaluation methodologies,
our research seeks to advance our understanding of these models' capabilities.
We propose a novel VQA benchmark based on well-known visual classification
datasets which allows a granular evaluation of text-generative vision-language
models and their comparison with discriminative vision-language models. To
improve the assessment of coarse answers on fine-grained classification tasks,
we suggest using the semantic hierarchy of the label space to ask automatically
generated follow-up questions about the ground-truth category. Finally, we
compare traditional NLP and LLM-based metrics for the problem of evaluating
model predictions given ground-truth answers. We perform a human evaluation
study upon which we base our decision on the final metric. We apply our
benchmark to a suite of vision-language models and show a detailed comparison
of their abilities on object, action, and attribute classification. Our
contributions aim to lay the foundation for more precise and meaningful
assessments, facilitating targeted progress in the exciting field of
vision-language modeling.

通过提出创新的评估方法并修正现有的视觉问答基准，我们的研究旨在推进我们对文本生成视觉语言模型能力的理解，提出了一种基于著名视觉分类数据集的新型视觉问答基准，可以对文本生成视觉语言模型进行细粒度评估，并与辨别性视觉语言模型进行比较。我们建议利用标签空间的语义层次结构为细粒度分类任务中的粗略答案提出自动生成的后续问题，以改善评估模型预测的传统自然语言处理和基于语言模型的度量标准。我们进行了人工评估研究，基于这项研究，我们决定采用最终的度量标准。我们将我们的基准应用于一套视觉语言模型，并详细比较了它们在对象、行为和属性分类方面的能力。我们的研究为更精确、有意义的评估奠定基础，促进了视觉语言建模领域的有针对性进展。