As Artificial General Intelligence (AGI) becomes increasingly integrated into
various facets of human life, ensuring the safety and ethical alignment of such
systems is paramount. Previous studies primarily focus on single-modality
threats, which may not suffice given the integrated and complex nature of
cross-modality interactions. We introduce a novel safety alignment challenge
called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety
alignment. Specifically, it considers cases where single modalities are safe
independently but could potentially lead to unsafe or unethical outputs when
combined. To empirically investigate this problem, we developed the SIUO, a
cross-modality benchmark encompassing 9 critical safety domains, such as
self-harm, illegal activities, and privacy violations. Our findings reveal
substantial safety vulnerabilities in both closed- and open-source LVLMs, such
as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably
interpret and respond to complex, real-world scenarios.

人工通用智能（AGI）越来越多地融入到人类生活的各个方面，确保这些系统的安全和道德一致性至关重要。过去的研究主要关注单模态威胁，这可能不足以应对交叉模态相互作用的综合和复杂性。我们提出了一个名为 “安全输入但不安全输出（SIUO）” 的新型安全对齐挑战，以评估交叉模态的安全对齐性。具体而言，它考虑了单一模态在独立情况下安全，但在组合时可能导致不安全或不道德的输出的情况。为了对这个问题进行实证调查，我们开发了 SIUO，一个涵盖了 9 个关键安全领域（如自残、非法活动和隐私侵犯）的交叉模态基准。我们的研究结果揭示了闭源和开源 LVLM（如 GPT-4V 和 LLaVA）在安全性上存在重大漏洞，凸显了当前模型在可靠解读和应对复杂的现实情景方面的不足。