Despite the remarkable performance of text-to-image diffusion models in image
generation tasks, recent studies have raised the issue that generated images
sometimes cannot capture the intended semantic contents of the text prompts,
which phenomenon is often called semantic misalignment. To address this, here
we present a novel energy-based model (EBM) framework. Specifically, we first
formulate EBMs of latent image representations and text embeddings in each
cross-attention layer of the denoising autoencoder. Then, we obtain the
gradient of the log posterior of context vectors, which can be updated and
transferred to the subsequent cross-attention layer, thereby implicitly
minimizing a nested hierarchy of energy functions. Our latent EBMs further
allow zero-shot compositional generation as a linear combination of
cross-attention outputs from different contexts. Using extensive experiments,
we demonstrate that the proposed method is highly effective in handling various
image generation tasks, including multi-concept generation, text-guided image
inpainting, and real and synthetic image editing.

本文提出了基于能量的模型框架，该框架可应用于图像生成任务中对文本提示的意义内容进行相应的处理。经过充分实验表明，该方法在处理多概念生成、文本引导的图像修补以及实际和人造图像编辑等各种图像生成任务方面非常有效。

文本到图像扩散模型中的贝叶斯上下文更新能量交叉注意力

Energy-Based Cross Attention for Bayesian Context Update in  Text-to-Image Diffusion Models

Latent image representations arising from vision-language models have proved
immensely useful for a variety of downstream tasks. However, their utility is
limited by their entanglement with respect to different visual attributes. For
instance, recent work has shown that CLIP image representations are often
biased toward specific visual properties (such as objects or actions) in an
unpredictable manner. In this paper, we propose to separate representations of
the different visual modalities in CLIP's joint vision-language space by
leveraging the association between parts of speech and specific visual modes of
variation (e.g. nouns relate to objects, adjectives describe appearance). This
is achieved by formulating an appropriate component analysis model that learns
subspaces capturing variability corresponding to a specific part of speech,
while jointly minimising variability to the rest. Such a subspace yields
disentangled representations of the different visual properties of an image or
text in closed form while respecting the underlying geometry of the manifold on
which the representations lie. What's more, we show the proposed model
additionally facilitates learning subspaces corresponding to specific visual
appearances (e.g. artists' painting styles), which enables the selective
removal of entire visual themes from CLIP-based text-to-image synthesis. We
validate the model both qualitatively, by visualising the subspace projections
with a text-to-image model and by preventing the imitation of artists' styles,
and quantitatively, through class invariance metrics and improvements to
baseline zero-shot classification. Our code is available at:
this https URL

本文提出了一种通过语法组件分解学习 CLIP 模型中视觉 - 语言联合空间不同视觉模态的基本变化，从而获得解耦的图像和模态表示的方法，并证明其在图像合成和零样本分类方面的有效性。