Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
across diverse tasks, they still face significant trustworthiness challenges.
Yet, current literature on the assessment of trustworthy MLLMs remains limited,
lacking a holistic evaluation to offer thorough insights into future
improvements. In this work, we establish MultiTrust, the first comprehensive
and unified benchmark on the trustworthiness of MLLMs across five primary
aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
employs a rigorous evaluation strategy that addresses both multimodal risks and
cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
Extensive experiments with 21 modern MLLMs reveal some previously unexplored
trustworthiness issues and risks, highlighting the complexities introduced by
the multimodality and underscoring the necessity for advanced methodologies to
enhance their reliability. For instance, typical proprietary models still
struggle with the perception of visually confusing images and are vulnerable to
multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
disclose privacy in text and reveal ideological and cultural biases even when
paired with irrelevant images in inference, indicating that the multimodality
amplifies the internal risks from base LLMs. Additionally, we release a
scalable toolbox for standardized trustworthiness research, aiming to
facilitate future advancements in this important field. Code and resources are
publicly available at: this https URL

通过建立 MultiTrust 综合基准，本研究通过对 21 种现代多模态大型语言模型的广泛实验，揭示了一些之前未开发的可信度问题与风险，强调了多模态性引入的复杂性，并强调提升其可靠性的先进方法的必要性。