The success of the reward model in distinguishing between responses with
subtle safety differences depends critically on the high-quality preference
dataset, which should capture the fine-grained nuances of harmful and harmless
responses. This motivates the need to develop a dataset involving preference
margins, which accurately quantify how harmless one response is compared to
another. In this paper, we take the first step to propose an effective and
cost-efficient framework to promote the margin-enhanced preference dataset
development. Our framework, Legend, Leverages representation engineering to
annotate preference datasets. It constructs the specific direction within the
LLM's embedding space that represents safety. By leveraging this safety
direction, Legend can then leverage the semantic distances of paired responses
along this direction to annotate margins automatically. We experimentally
demonstrate our effectiveness in both reward modeling and harmless alignment
for LLMs. Legend also stands out for its efficiency, requiring only the
inference time rather than additional training. This efficiency allows for
easier implementation and scalability, making Legend particularly valuable for
practical applications in aligning LLMs with safe conversations.

我们提出了一种有效且高效的框架来推动边界增强的偏好数据集的发展，通过利用语义距离对配对响应沿着安全方向进行自动标注，并在奖励建模和安全对齐方面进行了实验验证。