Recent alignment algorithms such as direct preference optimization (DPO) have been developed to improve the safety of large language models (LLMs) by training these models to match human behaviors exemplified by preference data. However, these methods are both computationally intensive and lacking in controllability and transparency, making them prone to jailbreaking and inhibiting their widespread use. Furthermore, these tuning-based methods require large-scale preference data for training and are susceptible to noisy preference data. In this paper, we introduce a tuning-free alignment alternative (DeTox) and demonstrate its effectiveness under the use case of toxicity reduction. Grounded on theory from factor analysis, DeTox is a sample-efficient model editing approach that identifies a toxic subspace in the model parameter space and reduces model toxicity by projecting away the detected subspace. The toxic sub-space is identified by extracting preference data embeddings from the language model, and removing non-toxic information from these embeddings. We show that DeTox is more sample-efficient than DPO, further showcasing greater robustness to noisy data. Finally, we establish both theoretical and empirical connections between DeTox and DPO, showing that DeTox can be interpreted as a denoised version of a single DPO step.

通过因子分析理论，引入无需调优的对齐替代方案（DeTox），该方案通过从模型的参数空间中检测出有毒子空间，通过投影方式去除检测到的子空间，从而减少模型的有毒性。我们证明了DeTox比DPO更加高效，并且对噪声数据具有更强的鲁棒性。最后，我们建立了DeTox和DPO之间的理论和实证关系，并展示了DeTox可以被解释为单一DPO步骤的去噪版本。

DeTox: 模型编辑的有毒子空间投影