Existing work on jailbreak Multimodal Large Language Models (MLLMs) has
focused primarily on adversarial examples in model inputs, with less attention
to vulnerabilities in model APIs. To fill the research gap, we carry out the
following work: 1) We discover a system prompt leakage vulnerability in GPT-4V.
Through carefully designed dialogue, we successfully steal the internal system
prompts of GPT-4V. This finding indicates potential exploitable security risks
in MLLMs; 2)Based on the acquired system prompts, we propose a novel MLLM
jailbreaking attack method termed SASP (Self-Adversarial Attack via System
Prompt). By employing GPT-4 as a red teaming tool against itself, we aim to
search for potential jailbreak prompts leveraging stolen system prompts.
Furthermore, in pursuit of better performance, we also add human modification
based on GPT-4's analysis, which further improves the attack success rate to
98.7\%; 3) We evaluated the effect of modifying system prompts to defend
against jailbreaking attacks. Results show that appropriately designed system
prompts can significantly reduce jailbreak success rates. Overall, our work
provides new insights into enhancing MLLM security, demonstrating the important
role of system prompts in jailbreaking, which could be leveraged to greatly
facilitate jailbreak success rates while also holding the potential for
defending against jailbreaks.

通过从 GPT-4V 中成功窃取内部系统提示并使用 SASP（自我对抗攻击系统提示）方法进行监狱破解，本研究发现了 Multimodal Large Language Models（MLLMs）的潜在安全风险并提出了相应的防御方法，以显著降低监狱破解的成功率。