The safety alignment of current Large Language Models (LLMs) is vulnerable.
Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned
models. We argue that many of these vulnerabilities are related to a shared
underlying issue: safety alignment can take shortcuts, wherein the alignment
adapts a model's generative distribution primarily over only its very first few
output tokens. We refer to this issue as shallow safety alignment. In this
paper, we present case studies to explain why shallow safety alignment can
exist and provide evidence that current aligned LLMs are subject to this issue.
We also show how these findings help explain multiple recently discovered
vulnerabilities in LLMs, including the susceptibility to adversarial suffix
attacks, prefilling attacks, decoding parameter attacks, and fine-tuning
attacks. Importantly, we discuss how this consolidated notion of shallow safety
alignment sheds light on promising research directions for mitigating these
vulnerabilities. For instance, we show that deepening the safety alignment
beyond just the first few tokens can often meaningfully improve robustness
against some common exploits. Finally, we design a regularized finetuning
objective that makes the safety alignment more persistent against fine-tuning
attacks by constraining updates on initial tokens. Overall, we advocate that
future safety alignment should be made more than just a few tokens deep.

当前大型语言模型（LLM）存在的安全对齐问题容易受到攻击，我们称之为浅安全对齐问题。本文通过案例研究解释了为什么浅安全对齐问题存在，并提供证据表明目前的安全对齐 LLMs 受到此问题的影响。我们还展示了这些发现如何帮助解释最近发现的 LLMs 的多个漏洞，包括对敌对性后缀攻击、填充攻击、解码参数攻击和微调攻击的敏感性。同时，我们讨论了浅安全对齐的综合概念如何为减轻这些漏洞指明了有价值的研究方向，并提出了一种通过限制对初始标记的更新来使安全对齐更具持久性的正则化微调目标。总之，我们主张未来的安全对齐应该超越前几个标记而更加深入。