As the number of large language models (LLMs) released to the public grows,
there is a pressing need to understand the safety implications associated with
these models learning from third-party custom finetuning data. We explore the
behavior of LLMs finetuned on noisy custom data containing unsafe content,
represented by datasets that contain biases, toxicity, and harmfulness, finding
that while aligned LLMs can readily learn this unsafe content, they also tend
to forget it more significantly than other examples when subsequently finetuned
on safer content. Drawing inspiration from the discrepancies in forgetting, we
introduce the "ForgetFilter" algorithm, which filters unsafe data based on how
strong the model's forgetting signal is for that data. We demonstrate that the
ForgetFilter algorithm ensures safety in customized finetuning without
compromising downstream task performance, unlike sequential safety finetuning.
ForgetFilter outperforms alternative strategies like replay and moral
self-correction in curbing LLMs' ability to assimilate unsafe content during
custom finetuning, e.g. 75% lower than not applying any safety measures and 62%
lower than using self-correction in toxicity score.

即使在第三方定制微调数据中学习不安全内容，大型语言模型（LLMs）也可以采用 “遗忘过滤器” 算法过滤不安全的数据，确保安全，同时不影响后续任务性能。