Classifier-free guidance (CFG) is a fundamental tool in modern diffusion
models for text-guided generation. Although effective, CFG has notable
drawbacks. For instance, DDIM with CFG lacks invertibility, complicating image
editing; furthermore, high guidance scales, essential for high-quality outputs,
frequently result in issues like mode collapse. Contrary to the widespread
belief that these are inherent limitations of diffusion models, this paper
reveals that the problems actually stem from the off-manifold phenomenon
associated with CFG, rather than the diffusion models themselves. More
specifically, inspired by the recent advancements of diffusion model-based
inverse problem solvers (DIS), we reformulate text-guidance as an inverse
problem with a text-conditioned score matching loss, and develop CFG++, a novel
approach that tackles the off-manifold challenges inherent in traditional CFG.
CFG++ features a surprisingly simple fix to CFG, yet it offers significant
improvements, including better sample quality for text-to-image generation,
invertibility, smaller guidance scales, reduced mode collapse, etc.
Furthermore, CFG++ enables seamless interpolation between unconditional and
conditional sampling at lower guidance scales, consistently outperforming
traditional CFG at all scales. Experimental results confirm that our method
significantly enhances performance in text-to-image generation, DDIM inversion,
editing, and solving inverse problems, suggesting a wide-ranging impact and
potential applications in various fields that utilize text guidance. Project
Page: this https URL

通过将文本引导问题重新定义为一个逆问题，并使用基于扩散模型的逆问题求解器，本文提出了一种名为 CFG++ 的新方法来解决传统 CFG 中存在的问题，包括改善文本到图像生成的样本质量、可逆性、减少模式崩溃等方面的显著改进，同时在所有规模上始终优于传统 CFG。实验证实了我们的方法在文本到图像生成、DDIM 反演、编辑和求解逆问题方面显著提高了性能，表明其在利用文本引导的各个领域中具有广泛的影响和潜在应用。

CFG++：约束流形的无监督扩散模型分类器引导

CFG++: Manifold-constrained Classifier Free Guidance for Diffusion  Models

We propose a new method for solving imaging inverse problems using
text-to-image latent diffusion models as general priors. Existing methods using
latent diffusion models for inverse problems typically rely on simple null text
prompts, which can lead to suboptimal performance. To address this limitation,
we introduce a method for prompt tuning, which jointly optimizes the text
embedding on-the-fly while running the reverse diffusion process. This allows
us to generate images that are more faithful to the diffusion prior. In
addition, we propose a method to keep the evolution of latent variables within
the range space of the encoder, by projection. This helps to reduce image
artifacts, a major problem when using latent diffusion models instead of
pixel-based diffusion models. Our combined method, called P2L, outperforms both
image- and latent-diffusion model-based inverse problem solvers on a variety of
tasks, such as super-resolution, deblurring, and inpainting.

我们提出了一种使用文本到图像潜在扩散模型作为通用先验来解决图像逆问题的新方法。我们引入了一种 prompt 调整方法，通过在运行反向扩散过程时动态优化文本嵌入，使得我们能够生成更符合扩散先验的图像。此外，我们提出了一种投影方法，以保持潜在变量在编码器的范围空间内的演化，从而有助于减少图像伪影问题。我们的综合方法 P2L 在各种任务（如超分辨率、去模糊和修复缺失部分）上优于基于图像和潜在扩散模型的逆问题求解器。