Recently, Direct Preference Optimization (DPO) has extended its success from
aligning large language models (LLMs) to aligning text-to-image diffusion
models with human preferences. Unlike most existing DPO methods that assume all
diffusion steps share a consistent preference order with the final generated
images, we argue that this assumption neglects step-specific denoising
performance and that preference labels should be tailored to each step's
contribution. To address this limitation, we propose Step-aware Preference
Optimization (SPO), a novel post-training approach that independently evaluates
and adjusts the denoising performance at each step, using a step-aware
preference model and a step-wise resampler to ensure accurate step-aware
supervision. Specifically, at each denoising step, we sample a pool of images,
find a suitable win-lose pair, and, most importantly, randomly select a single
image from the pool to initialize the next denoising step. This step-wise
resampler process ensures the next win-lose image pair comes from the same
image, making the win-lose comparison independent of the previous step. To
assess the preferences at each step, we train a separate step-aware preference
model that can be applied to both noisy and clean images. Our experiments with
Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms
the latest Diffusion-DPO in aligning generated images with complex, detailed
prompts and enhancing aesthetics, while also achieving more than 20x times
faster in training efficiency. Code and model:
this https URL

我们提出了一种 Step-aware Preference Optimization (SPO) 的方法，通过独立评估和调整每个步骤的去噪性能，在对生成图像进行偏好调整时改进了 Diffusion-DPO 方法的训练效率和图像质量。

步骤感知的偏好优化：在每个步骤中将偏好与去噪性能对齐

Step-aware Preference Optimization: Aligning Preference with Denoising  Performance at Each Step

Large language models (LLMs) are fine-tuned using human comparison data with
Reinforcement Learning from Human Feedback (RLHF) methods to make them better
aligned with users' preferences. In contrast to LLMs, human preference learning
has not been widely explored in text-to-image diffusion models; the best
existing approach is to fine-tune a pretrained model using carefully curated
high quality images and captions to improve visual appeal and text alignment.
We propose Diffusion-DPO, a method to align diffusion models to human
preferences by directly optimizing on human comparison data. Diffusion-DPO is
adapted from the recently developed Direct Preference Optimization (DPO), a
simpler alternative to RLHF which directly optimizes a policy that best
satisfies human preferences under a classification objective. We re-formulate
DPO to account for a diffusion model notion of likelihood, utilizing the
evidence lower bound to derive a differentiable objective. Using the Pick-a-Pic
dataset of 851K crowdsourced pairwise preferences, we fine-tune the base model
of the state-of-the-art Stable Diffusion XL (SDXL)-1.0 model with
Diffusion-DPO. Our fine-tuned base model significantly outperforms both base
SDXL-1.0 and the larger SDXL-1.0 model consisting of an additional refinement
model in human evaluation, improving visual appeal and prompt alignment. We
also develop a variant that uses AI feedback and has comparable performance to
training on human preferences, opening the door for scaling of diffusion model
alignment methods.

利用人类比较数据和强化学习通过 Reinforcement Learning from Human Feedback (RLHF) 方法对大规模语言模型（LLMs）进行微调，以使其更好地与用户偏好相匹配。与 LLMs 相比，文本到图像扩散模型中人类偏好学习的探索较少；目前最佳方法是使用经过精心筛选的高质量图像和标题对预训练模型进行微调，以提高视觉吸引力和文本对齐性。我们提出 Diffusion-DPO 方法，通过在人类比较数据上进行直接优化，使扩散模型与人类偏好相匹配。Diffusion-DPO 从最近开发的直接偏好优化（DPO）中进行适应，DPO 是对最佳满足分类目标下人类偏好的策略直接进行优化的更简单的方法。我们重新制定 DPO 以考虑扩散模型的似然概念，利用证据下界导出可微分的目标函数。利用 Pick-a-Pic 数据集中的 851K 个众包成对偏好，我们使用 Diffusion-DPO 对最先进的稳定扩散 XL（SDXL）-1.0 模型的基础模型进行微调。我们微调后的基础模型在人工评估中显著优于基础 SDXL-1.0 模型和额外的改进模型，从而提高了视觉吸引力和提示对齐。我们还开发了一个使用 AI 反馈并具有与基于人类偏好训练相当性能的变体，为扩展扩散模型对齐方法打开了大门。