Optimizing large language models (LLMs) for downstream use cases often
involves the customization of pre-trained LLMs through further fine-tuning.
Meta's open release of Llama models and OpenAI's APIs for fine-tuning GPT-3.5
Turbo on custom datasets also encourage this practice. But, what are the safety
costs associated with such custom fine-tuning? We note that while existing
safety alignment infrastructures can restrict harmful behaviors of LLMs at
inference time, they do not cover safety risks when fine-tuning privileges are
extended to end-users. Our red teaming studies find that the safety alignment
of LLMs can be compromised by fine-tuning with only a few adversarially
designed training examples. For instance, we jailbreak GPT-3.5 Turbo's safety
guardrails by fine-tuning it on only 10 such examples at a cost of less than
$0.20 via OpenAI's APIs, making the model responsive to nearly any harmful
instructions. Disconcertingly, our research also reveals that, even without
malicious intent, simply fine-tuning with benign and commonly used datasets can
also inadvertently degrade the safety alignment of LLMs, though to a lesser
extent. These findings suggest that fine-tuning aligned LLMs introduces new
safety risks that current safety infrastructures fall short of addressing --
even if a model's initial safety alignment is impeccable, it is not necessarily
to be maintained after custom fine-tuning. We outline and critically analyze
potential mitigations and advocate for further research efforts toward
reinforcing safety protocols for the custom fine-tuning of aligned LLMs.

通过细调大型语言模型 (LLMs) 进行定制以优化下游应用通常需要进一步在预训练的 LLMs 上进行微调。然而，此类自定义微调的安全成本是多少？我们的研究发现，尽管现有的安全对齐基础设施可以在推理时限制 LLMs 的有害行为，但当将微调权限扩展给最终用户时，它们却无法覆盖安全风险。我们的红队研究发现，只需使用少数恶意设计的训练样例对 GPT-3.5 Turbo 进行微调，就可能危及 LLMs 的安全对齐性。此外，我们的研究还揭示，即使没有恶意意图，只需使用良性且常用的数据集对 LLMs 进行微调，也可能无意中降低其安全对齐性。这些发现表明，细调对齐的 LLMs 引入了新的安全风险，而当前的安全基础设施无法很好地解决这些风险。我们概述并对潜在减轻措施进行了批判性分析，并倡导进一步的研究努力，以加强对齐的 LLMs 的自定义微调的安全协议。