Recent studies reveal that integrating new modalities into Large Language
Models (LLMs), such as Vision-Language Models (VLMs), creates a new attack
surface that bypasses existing safety training techniques like Supervised
Fine-tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF). While
further SFT and RLHF-based safety training can be conducted in multi-modal
settings, collecting multi-modal training datasets poses a significant
challenge. Inspired by the structural design of recent multi-modal models,
where, regardless of the combination of input modalities, all inputs are
ultimately fused into the language space, we aim to explore whether unlearning
solely in the textual domain can be effective for cross-modality safety
alignment. Our evaluation across six datasets empirically demonstrates the
transferability -- textual unlearning in VLMs significantly reduces the Attack
Success Rate (ASR) to less than 8\% and in some cases, even as low as nearly
2\% for both text-based and vision-text-based attacks, alongside preserving the
utility. Moreover, our experiments show that unlearning with a multi-modal
dataset offers no potential benefits but incurs significantly increased
computational demands, possibly up to 6 times higher.

将新的模态集成到大型语言模型（LLMs）中，如视觉 - 语言模型（VLMs），在绕过现有的安全训练技术（如 SFT 和 RLHF）的同时创造了一个新的攻击面。我们通过在文本领域进行反学习来实现跨模态安全对齐，实验证明在 VLMs 中进行文本反学习显著减少攻击成功率（ASR）至少低于 8％，甚至在某些情况下低至近 2％，同时保留实用性。