Instruction tuning enhances large vision-language models (LVLMs) but raises
security risks through potential backdoor attacks due to their openness.
Previous backdoor studies focus on enclosed scenarios with consistent training
and testing instructions, neglecting the practical domain gaps that could
affect attack effectiveness. This paper empirically examines the
generalizability of backdoor attacks during the instruction tuning of LVLMs for
the first time, revealing certain limitations of most backdoor strategies in
practical scenarios. We quantitatively evaluate the generalizability of six
typical backdoor attacks on image caption benchmarks across multiple LVLMs,
considering both visual and textual domain offsets. Our findings indicate that
attack generalizability is positively correlated with the backdoor trigger's
irrelevance to specific images/models and the preferential correlation of the
trigger pattern. Additionally, we modify existing backdoor attacks based on the
above key observations, demonstrating significant improvements in cross-domain
scenario generalizability (+86% attack success rate). Notably, even without
access to the instruction datasets, a multimodal instruction set can be
successfully poisoned with a very low poisoning rate (0.2%), achieving an
attack success rate of over 97%. This paper underscores that even simple
traditional backdoor strategies pose a serious threat to LVLMs, necessitating
more attention and in-depth research.

使用指令调优增强大规模视觉语言模型 (LVLMs) 会提高安全风险，因其开放性可能导致后门攻击。本研究首次经验性地考察了指令调优 LVLMs 期间后门攻击的普适性，揭示了在实际场景中大多数后门策略的某些限制。通过定量评估对视觉和文本领域存在偏差的六种典型后门攻击在图像字幕基准测试上的普适性，我们的研究结果表明，攻击的普适性与后门触发器与特定图像 / 模型的不相关性以及触发器模式的偏好相关。此外，我们基于以上关键观察修改了现有的后门攻击方法，在跨域场景的普适性方面取得了显著改进 (+86% 攻击成功率)。值得注意的是，即使没有访问指令数据集，也可以使用极低的污染率 (0.2%) 成功毒化多模态指令集，攻击成功率超过 97%。本研究强调即使是简单的传统后门策略也对 LVLMs 构成严重威胁，需要更多关注和深入研究。