In this work, we introduce the PKU-SafeRLHF dataset, designed to promote
research on safety alignment in large language models (LLMs). As a sibling
project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and
harmlessness for question-answering pairs, providing distinct perspectives on
these coupled attributes. Overall, we provide 44.6k refined prompts and 265k
question-answer pairs with safety meta-labels for 19 harm categories and three
severity levels ranging from minor to severe, with answers generated by
Llama-family models. Based on this, we collected 166.8k preference data,
including dual-preference (helpfulness and harmlessness decoupled) and
single-preference data (trade-off the helpfulness and harmlessness from
scratch), respectively. Using the large-scale annotation data, we further train
severity-sensitive moderation for the risk control of LLMs and safety-centric
RLHF algorithms for the safety alignment of LLMs. We believe this dataset will
be a valuable resource for the community, aiding in the safe deployment of
LLMs.

本文介绍了 PKU-SafeRLHF 数据集，旨在促进大型语言模型（LLMs）中的安全任务对齐研究。数据集包含 44.6k 个精细设定的提示和 265k 个问题 - 答案对，涵盖了 19 个伤害类别和从轻微到严重的三个严重程度级别，答案由 Llama 家族模型生成。利用大规模标注数据，我们进一步训练了对 LLMs 的风险控制的严重程度敏感的算法以及对 LLMs 的安全任务对齐的安全中心 RLHF 算法。我们相信这个数据集将成为社区宝贵的资源，有助于 LLMs 的安全部署。