Human visual imagination usually begins with analogies or rough sketches. For
example, given an image with a girl playing guitar before a building, one may
analogously imagine how it seems like if Iron Man playing guitar before Pyramid
in Egypt. Nonetheless, visual condition may not be precisely aligned with the
imaginary result indicated by text prompt, and existing layout-controllable
text-to-image (T2I) generation models is prone to producing degraded generated
results with obvious artifacts. To address this issue, we present a novel T2I
generation method dubbed SmartControl, which is designed to modify the rough
visual conditions for adapting to text prompt. The key idea of our SmartControl
is to relax the visual condition on the areas that are conflicted with text
prompts. In specific, a Control Scale Predictor (CSP) is designed to identify
the conflict regions and predict the local control scales, while a dataset with
text prompts and rough visual conditions is constructed for training CSP. It is
worth noting that, even with a limited number (e.g., 1,000~2,000) of training
samples, our SmartControl can generalize well to unseen objects. Extensive
experiments on four typical visual condition types clearly show the efficacy of
our SmartControl against state-of-the-arts. Source code, pre-trained models,
and datasets are available at this https URL

通过设计一种名为 SmartControl 的新型 T2I 生成方法，我们解决了现有 T2I 生成模型中存在的问题，该方法通过放松与文本提示冲突的区域的视觉条件，从而调整图像的粗略视觉条件，使之与文本提示相适应，并在四种典型的视觉条件类型上充分验证了我们的 SmartControl 的有效性。

SmartControl：增强控制网络以处理复杂视觉环境

SmartControl: Enhancing ControlNet for Handling Rough Visual Conditions

Achieving machine autonomy and human control often represent divergent
objectives in the design of interactive AI systems. Visual generative
foundation models such as Stable Diffusion show promise in navigating these
goals, especially when prompted with arbitrary languages. However, they often
fall short in generating images with spatial, structural, or geometric
controls. The integration of such controls, which can accommodate various
visual conditions in a single unified model, remains an unaddressed challenge.
In response, we introduce UniControl, a new generative foundation model that
consolidates a wide array of controllable condition-to-image (C2I) tasks within
a singular framework, while still allowing for arbitrary language prompts.
UniControl enables pixel-level-precise image generation, where visual
conditions primarily influence the generated structures and language prompts
guide the style and context. To equip UniControl with the capacity to handle
diverse visual conditions, we augment pretrained text-to-image diffusion models
and introduce a task-aware HyperNet to modulate the diffusion models, enabling
the adaptation to different C2I tasks simultaneously. Trained on nine unique
C2I tasks, UniControl demonstrates impressive zero-shot generation abilities
with unseen visual conditions. Experimental results show that UniControl often
surpasses the performance of single-task-controlled methods of comparable model
sizes. This control versatility positions UniControl as a significant
advancement in the realm of controllable visual generation.

UniControl 是一种新的可控生成基础模型，集成了多种可控因素，实现了基于任意语言提示的像素级精确图像生成，并通过多任务学习，使其具有适应不同可控条件的能力，实验证明其性能优于其他同类型模型。