Human visual imagination usually begins with analogies or rough sketches. For
example, given an image with a girl playing guitar before a building, one may
analogously imagine how it seems like if Iron Man playing guitar before Pyramid
in Egypt. Nonetheless, visual condition may not be precisely aligned with the
imaginary result indicated by text prompt, and existing layout-controllable
text-to-image (T2I) generation models is prone to producing degraded generated
results with obvious artifacts. To address this issue, we present a novel T2I
generation method dubbed SmartControl, which is designed to modify the rough
visual conditions for adapting to text prompt. The key idea of our SmartControl
is to relax the visual condition on the areas that are conflicted with text
prompts. In specific, a Control Scale Predictor (CSP) is designed to identify
the conflict regions and predict the local control scales, while a dataset with
text prompts and rough visual conditions is constructed for training CSP. It is
worth noting that, even with a limited number (e.g., 1,000~2,000) of training
samples, our SmartControl can generalize well to unseen objects. Extensive
experiments on four typical visual condition types clearly show the efficacy of
our SmartControl against state-of-the-arts. Source code, pre-trained models,
and datasets are available at this https URL

通过设计一种名为 SmartControl 的新型 T2I 生成方法，我们解决了现有 T2I 生成模型中存在的问题，该方法通过放松与文本提示冲突的区域的视觉条件，从而调整图像的粗略视觉条件，使之与文本提示相适应，并在四种典型的视觉条件类型上充分验证了我们的 SmartControl 的有效性。