Public LLMs such as the Llama 2-Chat have driven huge activity in LLM
research. These models underwent alignment training and were considered safe.
Recently Qi et al. (2023) reported that even benign fine-tuning (e.g., on
seemingly safe datasets) can give rise to unsafe behaviors in the models. The
current paper is about methods and best practices to mitigate such loss of
alignment. Through extensive experiments on several chat models (Meta's Llama
2-Chat, Mistral AI's Mistral 7B Instruct v0.2, and OpenAI's GPT-3.5 Turbo),
this paper uncovers that the prompt templates used during fine-tuning and
inference play a crucial role in preserving safety alignment, and proposes the
"Pure Tuning, Safe Testing" (PTST) principle -- fine-tune models without a
safety prompt, but include it at test time. Fine-tuning experiments on GSM8K,
ChatDoctor, and OpenOrca show that PTST significantly reduces the rise of
unsafe behaviors, and even almost eliminates them in some cases.

本文研究了如何减轻模型由于微调引起的安全问题，通过对几个聊天模型进行广泛实验，发现在微调和推理过程中使用的提示模板对于保持安全对齐至关重要，并提出了 “纯微调，安全测试”（PTST）原则，即在没有安全提示的情况下微调模型，但在测试时使用它。在 GSM8K，ChatDoctor 和 OpenOrca 上进行的微调实验表明，PTST 显著减少了不安全行为的发生，甚至在某些情况下几乎消除了它们。