This paper introduces a novel and significant challenge for Vision Language
Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the
VLM's ability to withhold answers when faced with unsolvable problems in the
context of Visual Question Answering (VQA) tasks. UPD encompasses three
distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set
Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply
investigate the UPD problem, extensive experiments indicate that most VLMs,
including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying
extents, highlighting significant room for the improvements. To address UPD, we
explore both training-free and training-based solutions, offering new insights
into their effectiveness and limitations. We hope our insights, together with
future efforts within the proposed UPD settings, will enhance the broader
understanding and development of more practical and reliable VLMs.

本文介绍了一项名为无法解决问题检测（UPD）的视觉语言模型（VLM）的新挑战，通过视觉问答（VQA）任务中遇到无法解决的问题来考察 VLM 在保留答案方面的能力。我们通过广泛的实验发现，包括 GPT-4V 和 LLaVA-Next-34B 在内的大多数 VLM 都在不同程度上面临我们的基准测试的困难，凸显了改进的重要空间。为了解决 UPD 问题，我们探索了无需训练和基于训练的解决方案，为它们的有效性和局限性提供了新的见解。我们希望我们的见解，以及在所提出的 UPD 设置下的未来努力，能够增强对更实用和可靠的 VLM 的广泛理解和开发。