Text-to-image (T2I) diffusion models have become prominent tools for generating high-fidelity images from text prompts. However, when trained on unfiltered internet data, these models can produce unsafe, incorrect, or stylistically undesirable images that are not aligned with human preferences. To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences. Although effective, these methods are vulnerable to reward hacking, where the model overfits to the reward function, leading to a loss of diversity in the generated images. In this paper, we prove the inevitability of reward hacking and study natural regularization techniques like KL divergence and LoRA scaling, and their limitations for diffusion models. We also introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling, which retains the diversity of the base model while achieving Pareto-Optimal reward-diversity tradeoffs. Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity. Furthermore, a user study confirms that AIG improves diversity and quality of generated images across different model architectures and reward functions.

本研究针对文本到图像扩散模型在训练过程中产生的不安全和不符合人类偏好的图像问题，提出了一个新的解决方案。通过引入受启发的Annealed Importance Guidance (AIG) 正则化技术，我们在优化奖励的同时保留了生成图像的多样性，实验结果显示AIG可以有效提高图像的多样性和质量，从而实现奖励与多样性的最佳权衡。

阐明文本到图像扩散模型中的最佳奖励-多样性权衡