Visual Question Answering (VQA) entails answering questions about images. We
introduce the first VQA dataset in which all contents originate from an
authentic use case. Sourced from online question answering community forums, we
call it VQAonline. We then characterize our dataset and how it relates to eight
other VQA datasets. Observing that answers in our dataset tend to be much
longer (e.g., with a mean of 173 words) and thus incompatible with standard VQA
evaluation metrics, we next analyze which of the six popular metrics for longer
text evaluation align best with human judgments. We then use the best-suited
metrics to evaluate six state-of-the-art vision and language foundation models
on VQAonline and reveal where they struggle most. We will release the dataset
soon to facilitate future extensions.

VQAonline 是第一个所有内容都来源于真实用例的 VQA 数据集，该数据集中的答案往往较长，并且不适用于标准的 VQA 评估指标；研究发现，在长文本评估方面，六个流行的度量标准与人工判断相符度最高；使用最适合的度量标准评估六个最先进的视觉和语言基础模型在 VQAonline 上，揭示了它们最困难的问题。