Recent advancements in human preference optimization, initially developed for
Language Models (LMs), have shown promise for text-to-image Diffusion Models,
enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
Diffusion Models typically optimize in pixel or VAE space, which does not align
well with human perception, leading to slower and less efficient training
during the preference alignment stage. We propose using a perceptual objective
in the U-Net embedding space of the diffusion model to address these issues.
Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
supervised fine-tuning (SFT) within this embedding space. This method
significantly outperforms standard latent-space implementations across various
metrics, including quality and computational cost. For SDXL, our approach
provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
while significantly reducing compute. Our approach not only improves the
efficiency and quality of human preference alignment for diffusion models but
is also easily integrable with other optimization techniques. The training code
and LoRA weights will be available here:
this https URL

通过在扩散模型的内嵌空间中优化感知目标，我们提出了一种方法，使用直接偏好优化 (DPO)、对比偏好优化 (CPO) 和监督微调 (SFT) 来显著提高扩散模型的效率和质量，同时降低了计算成本。

用噪声条件化感知调整扩散模型

Aligning Diffusion Models with Noise-Conditioned Perception

Direct Preference Optimization (DPO), a standard method for aligning language
models with human preferences, is traditionally applied to offline preferences.
Recent studies show that DPO benefits from iterative training with online
preferences labeled by a trained reward model. In this work, we identify a
pitfall of vanilla iterative DPO - improved response quality can lead to
increased verbosity. To address this, we introduce iterative length-regularized
DPO (iLR-DPO) to penalize response length. Our empirical results show that
iLR-DPO can enhance a 7B model to perform on par with GPT-4 without increasing
verbosity. Specifically, our 7B model achieves a $50.5\%$ length-controlled win
rate against $\texttt{GPT-4 Preview}$ on AlpacaEval 2.0, and excels across
standard benchmarks including MT-Bench, Arena-Hard and OpenLLM Leaderboard.
These results demonstrate the effectiveness of iterative DPO in aligning
language models with human feedback.

我们引入了迭代的长度正则化 DPO（iLR-DPO）来解决改进的回应质量可能导致冗长的问题，并验证了 iLR-DPO 在与 GPT-4 相媲美的性能上的有效性。