Large vision-language models (LVLMs) have recently achieved rapid progress,
exhibiting great perception and reasoning abilities concerning visual
information. However, when faced with prompts in different sizes of solution
spaces, LVLMs fail to always give consistent answers regarding the same
knowledge point. This inconsistency of answers between different solution
spaces is prevalent in LVLMs and erodes trust. To this end, we provide a
multi-modal benchmark ConBench, to intuitively analyze how LVLMs perform when
the solution space of a prompt revolves around a knowledge point. Based on the
ConBench tool, we are the first to reveal the tapestry and get the following
findings: (1) In the discriminate realm, the larger the solution space of the
prompt, the lower the accuracy of the answers. (2) Establish the relationship
between the discriminative and generative realms: the accuracy of the
discriminative question type exhibits a strong positive correlation with its
Consistency with the caption. (3) Compared to open-source models, closed-source
models exhibit a pronounced bias advantage in terms of Consistency. Eventually,
we ameliorate the consistency of LVLMs by trigger-based diagnostic refinement,
indirectly improving the performance of their caption. We hope this paper will
accelerate the research community in better evaluating their models and
encourage future advancements in the consistency domain.

通过多模态基准测试工具 ConBench，本研究首次揭示了大型视觉和语言模型在解决方案空间不同的提示下的答案一致性问题，并通过基于触发器的诊断优化方法，间接提高了模型的性能，以增强其描述能力。