Teaching Visual Question Answering (VQA) models to abstain from unanswerable
questions is indispensable for building a trustworthy AI system. Existing
studies, though have explored various aspects of VQA, yet marginally ignored
this particular attribute. This paper aims to bridge the research gap by
contributing a comprehensive dataset, called UNK-VQA. The dataset is
specifically designed to address the challenge of questions that can be
unanswerable. To this end, we first augment the existing data via deliberate
perturbations on either the image or question. In specific, we carefully ensure
that the question-image semantics remain close to the original unperturbed
distribution. By means of this, the identification of unanswerable questions
becomes challenging, setting our dataset apart from others that involve mere
image replacement. We then extensively evaluate the zero- and few-shot
performance of several emerging multi-modal large models and discover
significant limitations of them when applied to our dataset. Additionally, we
also propose a straightforward method to tackle these unanswerable questions.
This dataset, we believe, will serve as a valuable benchmark for enhancing the
abstention capability of VQA models, thereby leading to increased
trustworthiness of AI systems.

本研究旨在构建一个值得信赖的 AI 系统，通过为 Visual Question Answering（VQA）模型教授弃权无法回答问题的能力。研究通过提供一个名为 UNK-VQA 的全面数据集来填补这一研究空白，该数据集特别设计用于解决无法回答的问题。同时，通过对图像或问题进行有意的扰动，充分评估了多模式大型模型的零或少样本性能，并提出了解决这些无法回答问题的简单方法。这一数据集将为提升 VQA 模型的弃权能力提供一个有价值的基准，从而增加 AI 系统的可信度。