The ability to collect a large dataset of human preferences from
text-to-image users is usually limited to companies, making such datasets
inaccessible to the public. To address this issue, we create a web app that
enables text-to-image users to generate images and specify their preferences.
Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image
prompts and real users' preferences over generated images. We leverage this
dataset to train a CLIP-based scoring function, PickScore, which exhibits
superhuman performance on the task of predicting human preferences. Then, we
test PickScore's ability to perform model evaluation and observe that it
correlates better with human rankings than other automatic evaluation metrics.
Therefore, we recommend using PickScore for evaluating future text-to-image
generation models, and using Pick-a-Pic prompts as a more relevant dataset than
MS-COCO. Finally, we demonstrate how PickScore can enhance existing
text-to-image models via ranking.

通过创建一个网络应用程序，我们制作了 Pick-a-Pic 数据集，以收集来自文本到图像用户的大规模偏好数据。使用该数据集，我们训练了一种基于 CLIP 的评分函数 PickScore，并证明其在预测人类偏好方面具有超人级的性能。因此，我们建议将 PickScore 用于评估未来的文本到图像生成模型，并使用 Pick-a-Pic 提示作为比 MS-COCO 更相关的数据集。最后，我们演示了如何通过排名来提高现有的文本到图像模型。