To mitigate the risk of harmful outputs from large vision models (LVMs), we
introduce the SafeSora dataset to promote research on aligning text-to-video
generation with human values. This dataset encompasses human preferences in
text-to-video generation tasks along two primary dimensions: helpfulness and
harmlessness. To capture in-depth human preferences and facilitate structured
reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and
harmlessness into 12 sub-categories, serving as the basis for pilot
annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique
videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations
labeled by humans. We further demonstrate the utility of the SafeSora dataset
through several applications, including training the text-video moderation
model and aligning LVMs with human preference by fine-tuning a prompt
augmentation module or the diffusion model. These applications highlight its
potential as the foundation for text-to-video alignment research, such as human
preference modeling and the development and validation of alignment algorithms.

为了减轻大规模视觉模型（LVMs）带来的有害输出的风险，我们引入了 SafeSora 数据集，促进文本到视频生成与人类价值观的对齐研究。该数据集包括帮助性和无害性两个主要维度上的人类偏好，并进一步细分为 4 个子维度和 12 个子类别。通过多个应用，我们展示了 SafeSora 数据集的实用性，例如培训文本 - 视频审查模型、通过对齐算法的微调来与人类偏好相对齐的 LVMs 等。这些应用突显了 SafeSora 数据集在文本到视频对齐研究中的潜力，例如人类偏好建模和对齐算法的开发与验证。