Detoxifying multilingual Large Language Models (LLMs) has become crucial due
to their increasing global use. In this work, we explore zero-shot
cross-lingual generalization of preference tuning in detoxifying LLMs. Unlike
previous studies that show limited cross-lingual generalization for other
safety tasks, we demonstrate that Direct Preference Optimization (DPO) training
with only English data can significantly reduce toxicity in multilingual
open-ended generations. For example, the probability of mGPT-1.3B generating
toxic continuations drops from 46.8% to 3.9% across 17 different languages
after training. Our results also extend to other multilingual LLMs, such as
BLOOM, Llama3, and Aya-23. Using mechanistic interpretability tools like causal
intervention and activation analysis, we identified the dual multilinguality
property of MLP layers in LLMs, which explains the cross-lingual generalization
of DPO. Finally, we show that bilingual sentence retrieval can predict the
cross-lingual transferability of DPO preference tuning.

通过对多语言大型语言模型进行解毒已经变得至关重要。本研究探讨了解毒语言模型中无监督跨语言泛化的偏好调整。通过仅使用英文数据进行直接优化训练，能够显著降低多语言开放生成中的有害信息。通过一系列实验证明，在训练后，mGPT-1.3B 生成有害连续内容的概率从 46.8% 降低到了 3.9%，涵盖了 17 种不同的语言。我们的结果也适用于其他多语言大型语言模型，如 BLOOM、Llama3 和 Aya-23。通过因果干预和激活分析等机械性可解释性工具，我们发现了多语言感知层在多语言大型语言模型中的双重特性，这解释了直接优化训练的跨语言泛化。最后，我们展示了双语句子检索可以预测 DPO 偏好调整的跨语言可迁移性。