Current Large Language Models (LLMs), even those tuned for safety and
alignment, are susceptible to jailbreaking. Some have found that just further
fine-tuning an aligned model with benign data (i.e., data without harmful
content) surprisingly leads to substantial degradation in safety. We delve into
the data-centric aspects of why benign fine-tuning inadvertently contributes to
jailbreaking. First, we represent fine-tuning data through two lenses:
representation and gradient spaces. Furthermore, we propose a bi-directional
anchoring method that prioritizes data points that are close to harmful
examples and distant from benign ones. By doing so, our approach effectively
identifies subsets of benign data that are more likely to degrade the model's
safety after fine-tuning. Training on just 100 of these seemingly benign
datapoints can lead to the fine-tuned model affirmatively responding to > 70%
of tested harmful requests, compared to < 20% after fine-tuning on randomly
selected data. We further find that selected data are often in the form of
lists and bullet points, or math questions.

通过双向锚定方法优先选择与有害示例接近但远离良性示例的数据点，我们的方法有效地识别了在微调后可能降低模型安全性的良性数据子集。