The alignment of large language models (LLMs) with human preferences remains a key challenge. While post-training techniques like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have achieved notable success, they often introduce computational inefficiencies and training instability. In this paper, we propose Feature-level constrained Preference Optimization (FPO), a novel method designed to simplify the alignment process while ensuring stability. FPO leverages pre-trained Sparse Autoencoders (SAEs) and introduces feature-level constraints, allowing for efficient, sparsity-enforced alignment. Our approach enjoys efficiency by using sparse features activated in a well-trained sparse autoencoder and the quality of sequential KL divergence by using the feature-level offline reference. Experimental results on benchmark datasets demonstrate that FPO achieves a 5.08% absolute improvement in win rate with much lower computational cost compared to state-of-the-art baselines, making it a promising solution for efficient and controllable LLM alignments.

本研究解决了大语言模型（LLM）与人类偏好对齐中的计算效率和训练不稳定性问题。提出的特征级约束偏好优化（FPO）方法通过引入特征级约束，利用预训练的稀疏自编码器（SAE），实现了高效和稳定的对齐。实验结果表明，FPO在赢率方面相对于最先进的基线实现了5.08%的绝对提升，同时大幅降低了计算成本，为高效可控的LLM对齐提供了有前景的解决方案。

使用稀疏特征级约束的直接偏好优化