Instruction tuning is an effective technique to align large language models
(LLMs) with human intents. In this work, we investigate how an adversary can
exploit instruction tuning by injecting specific instruction-following examples
into the training data that intentionally changes the model's behavior. For
example, an adversary can achieve content injection by injecting training
examples that mention target content and eliciting such behavior from
downstream models. To achieve this goal, we propose \textit{AutoPoison}, an
automated data poisoning pipeline. It naturally and coherently incorporates
versatile attack goals into poisoned data with the help of an oracle LLM. We
showcase two example attacks: content injection and over-refusal attacks, each
aiming to induce a specific exploitable behavior. We quantify and benchmark the
strength and the stealthiness of our data poisoning scheme. Our results show
that AutoPoison allows an adversary to change a model's behavior by poisoning
only a small fraction of data while maintaining a high level of stealthiness in
the poisoned examples. We hope our work sheds light on how data quality affects
the behavior of instruction-tuned models and raises awareness of the importance
of data quality for responsible deployments of LLMs. Code is available at
https://github.com/azshue/AutoPoison.

介绍了一种名为 AutoPoisin 的自动数据污染技术，可通过注入恶意训练数据使语言模型出现漏洞，并展示了其中的两种攻击：内容注入和过度拒绝攻击。