Humans are prone to cognitive distortions -- biased thinking patterns that
lead to exaggerated responses to specific stimuli, albeit in very different
contexts. This paper demonstrates that advanced Multimodal Large Language
Models (MLLMs) exhibit similar tendencies. While these models are designed to
respond queries under safety mechanism, they sometimes reject harmless queries
in the presence of certain visual stimuli, disregarding the benign nature of
their contexts. As the initial step in investigating this behavior, we identify
three types of stimuli that trigger the oversensitivity of existing MLLMs:
Exaggerated Risk, Negated Harm, and Counterintuitive Interpretation. To
systematically evaluate MLLMs' oversensitivity to these stimuli, we propose the
Multimodal OverSenSitivity Benchmark (MOSSBench). This toolkit consists of 300
manually collected benign multimodal queries, cross-verified by third-party
reviewers (AMT). Empirical studies using MOSSBench on 20 MLLMs reveal several
insights: (1). Oversensitivity is prevalent among SOTA MLLMs, with refusal
rates reaching up to 76% for harmless queries. (2). Safer models are more
oversensitive: increasing safety may inadvertently raise caution and
conservatism in the model's responses. (3). Different types of stimuli tend to
cause errors at specific stages -- perception, intent reasoning, and safety
judgement -- in the response process of MLLMs. These findings highlight the
need for refined safety mechanisms that balance caution with contextually
appropriate responses, improving the reliability of MLLMs in real-world
applications. We make our project available at
this https URL

通过研究发现，高级多模态大型语言模型（MLLMs）和人类一样容易出现认知失真，即在不同情境下对特定刺激作出夸大的反应。在研究中，我们识别出三类触发现有 MLLMs 过度敏感的刺激：夸大风险、否定伤害和违反直觉的解释。通过提出多模态过敏性基准 (MOSSBench) 来系统评估 MLLMs 对这些刺激的过度敏感性。实证研究发现，最先进的 MLLMs 存在过度敏感的问题，并且安全性越高的模型越容易出现敏感性。不同类型的刺激在 MLLMs 响应过程的感知、意图推理和安全判断阶段容易引发错误。这些发现突出了在现实世界应用中提高 MLLMs 可靠性的需求，需要更加精细的安全机制来平衡谨慎性和情境上适当的回应。