Diffusion models have exhibited impressive prowess in the text-to-image task.
Recent methods add image-level controls, e.g., edge and depth maps, to
manipulate the generation process together with text prompts to obtain desired
images. This controlling process is globally operated on the entire image,
which limits the flexibility of control regions. In this paper, we introduce a
new simple yet practical task setting: local control. It focuses on controlling
specific local areas according to user-defined image conditions, where the rest
areas are only conditioned by the original text prompt. This manner allows the
users to flexibly control the image generation in a fine-grained way. However,
it is non-trivial to achieve this goal. The naive manner of directly adding
local conditions may lead to the local control dominance problem. To mitigate
this problem, we propose a training-free method that leverages the updates of
noised latents and parameters in the cross-attention map during the denosing
process to promote concept generation in non-control areas. Moreover, we use
feature mask constraints to mitigate the degradation of synthesized image
quality caused by information differences inside and outside the local control
area. Extensive experiments demonstrate that our method can synthesize
high-quality images to the prompt under local control conditions. Code is
available at this https URL

本文介绍了一种新的简单而实用的任务设置：局部控制，它通过用户定义的图像条件在特定的局部区域进行控制，其余区域仅通过原始文本提示进行调节。我们提出了一种无需训练的方法，利用去噪过程中噪声潜变量和参数的更新，在交互注意力图中促进非控制区域的概念生成。此外，我们使用特征屏蔽约束解决局部控制区域内外信息差异导致的合成图像质量下降问题。广泛的实验表明，我们的方法可以在局部控制条件下合成出高质量的图像。

本地条件控制文本到图像扩散模型

Local Conditional Controlling for Text-to-Image Diffusion Models

Recently, there has been an increasing interest in automated prompt
optimization based on reinforcement learning (RL). This approach offers
important advantages, such as generating interpretable prompts and being
compatible with black-box foundation models. However, the substantial prompt
space size poses challenges for RL-based methods, often leading to suboptimal
policy convergence. This paper introduces MultiPrompter, a new framework that
views prompt optimization as a cooperative game between prompters which take
turns composing a prompt together. Our cooperative prompt optimization
effectively reduces the problem size and helps prompters learn optimal prompts.
We test our method on the text-to-image task and show its ability to generate
higher-quality images than baselines.

最近，基于强化学习的自动化提示优化引起了越来越多的关注。这种方法具有重要优势，比如生成可解释的提示并与黑匣子基础模型兼容。然而，庞大的提示空间大小对于基于强化学习的方法构成挑战，常常导致次优策略收敛。本文提出了 MultiPrompter，一个新的框架，将提示优化视为一种在协作博弈中，由提示者轮流共同组成提示的过程。我们的协作提示优化有效地减小了问题的规模，并帮助提示者学习到最优提示。我们在文本到图像任务上测试了我们的方法，并展示了其生成比基准模型更高质量图像的能力。