We present a novel image editing scenario termed Text-grounded Object
Generation (TOG), defined as generating a new object in the real image
spatially conditioned by textual descriptions. Existing diffusion models
exhibit limitations of spatial perception in complex real-world scenes, relying
on additional modalities to enforce constraints, and TOG imposes heightened
challenges on scene comprehension under the weak supervision of linguistic
information. We propose a universal framework ST-LDM based on Swin-Transformer,
which can be integrated into any latent diffusion model with training-free
backward guidance. ST-LDM encompasses a global-perceptual autoencoder with
adaptable compression scales and hierarchical visual features, parallel with
deformable multimodal transformer to generate region-wise guidance for the
subsequent denoising process. We transcend the limitation of traditional
attention mechanisms that only focus on existing visual features by introducing
deformable feature alignment to hierarchically refine spatial positioning fused
with multi-scale visual and linguistic information. Extensive Experiments
demonstrate that our model enhances the localization of attention mechanisms
while preserving the generative capabilities inherent to diffusion models.

我们提出了一种称为文本引导对象生成（TOG）的新型图像编辑场景，即在实际图像中通过文本描述在空间上生成一个新的对象。我们的模型基于 Swin-Transformer，具有全局感知自编码器和适应性压缩尺度以及分层视觉特征，用于生成下一个去噪过程的区域导向。通过引入可变形特征对齐来在融合多尺度视觉和语言信息的基础上层次性地优化空间定位，我们克服了传统注意机制只关注现有视觉特征的局限性。广泛的实验证明我们的模型在提高注意机制的定位能力的同时保留了扩散模型固有的生成能力。