Diffusion-based models have achieved state-of-the-art performance on
text-to-image synthesis tasks. However, one critical limitation of these models
is the low fidelity of generated images with respect to the text description,
such as missing objects, mismatched attributes, and mislocated objects. One key
reason for such inconsistencies is the inaccurate cross-attention to text in
both the spatial dimension, which controls at what pixel region an object
should appear, and the temporal dimension, which controls how different levels
of details are added through the denoising steps. In this paper, we propose a
new text-to-image algorithm that adds explicit control over spatial-temporal
cross-attention in diffusion models. We first utilize a layout predictor to
predict the pixel regions for objects mentioned in the text. We then impose
spatial attention control by combining the attention over the entire text
description and that over the local description of the particular object in the
corresponding pixel region of that object. The temporal attention control is
further added by allowing the combination weights to change at each denoising
step, and the combination weights are optimized to ensure high fidelity between
the image and the text. Experiments show that our method generates images with
higher fidelity compared to diffusion-model-based baselines without fine-tuning
the diffusion model. Our code is publicly available at
this https URL

本文提出了一种新的文本到图像算法，通过在扩散模型中加入显式的空间 - 时间交叉注意力控制，利用布局预测器和空间注意力控制相结合，实现优化组合权重的方式，从而生成与文本更高保真的图像。