Direct Preference Optimization (DPO) has been proposed as an effective and
efficient alternative to reinforcement learning from human feedback (RLHF). In
this paper, we propose a novel and enhanced version of DPO based on curriculum
learning for text-to-image generation. Our method is divided into two training
stages. First, a ranking of the examples generated for each prompt is obtained
by employing a reward model. Then, increasingly difficult pairs of examples are
sampled and provided to a text-to-image generative (diffusion or consistency)
model. Generated samples that are far apart in the ranking are considered to
form easy pairs, while those that are close in the ranking form hard pairs. In
other words, we use the rank difference between samples as a measure of
difficulty. The sampled pairs are split into batches according to their
difficulty levels, which are gradually used to train the generative model. Our
approach, Curriculum DPO, is compared against state-of-the-art fine-tuning
approaches on three benchmarks, outperforming the competing methods in terms of
text alignment, aesthetics and human preference. Our code is available at
this https URL

该论文介绍了一种基于课程学习的新颖增强版 Direct Preference Optimization（DPO）方法，用于文本到图像生成，在三个基准测试中胜过了其他方法，包括文本对齐、美学和人类首选项等方面。