Shape can specify key object constraints, yet existing text-to-image diffusion models ignore this cue and synthesize objects that are incorrectly scaled, cut off, or replaced with background content. We propose a training-free method, Shape-Guided Diffusion, which uses a novel Inside-Outside Attention mechanism to constrain the cross-attention (and self-attention) maps such that prompt tokens (and pixels) referring to the inside of the shape cannot attend outside the shape, and vice versa. To demonstrate the efficacy of our method, we propose a new image editing task where the model must replace an object specified by its mask and a text prompt. We curate a new ShapePrompts benchmark based on MS-COCO and achieve SOTA results in shape faithfulness, text alignment, and realism according to both quantitative metrics and human preferences. Our data and code will be made available at https://shape-guided-diffusion.github.io.

本研究提出了一种名为Shape-Guided Diffusion的新方法，使用Inside-Outside Attention机制，通过用户指定或自动推理的形状输入来修改预训练扩散模型，以应用形状限制于交叉和自注意图，并实现按文本提示替换对象的形状引导编辑任务。通过构建的MS-COCO数据集，本研究在形状忠诚度方面取得了SOTA结果。

基于形状导向内外注意力的扩散