The quality of the prompts provided to text-to-image diffusion models
determines how faithful the generated content is to the user's intent, often
requiring `prompt engineering'. To harness visual concepts from target images
without prompt engineering, current approaches largely rely on embedding
inversion by optimizing and then mapping them to pseudo-tokens. However,
working with such high-dimensional vector representations is challenging
because they lack semantics and interpretability, and only allow simple vector
operations when using them. Instead, this work focuses on inverting the
diffusion model to obtain interpretable language prompts directly. The
challenge of doing this lies in the fact that the resulting optimization
problem is fundamentally discrete and the space of prompts is exponentially
large; this makes using standard optimization techniques, such as stochastic
gradient descent, difficult. To this end, we utilize a delayed projection
scheme to optimize for prompts representative of the vocabulary space in the
model. Further, we leverage the findings that different timesteps of the
diffusion process cater to different levels of detail in an image. The later,
noisy, timesteps of the forward diffusion process correspond to the semantic
information, and therefore, prompt inversion in this range provides tokens
representative of the image semantics. We show that our approach can identify
semantically interpretable and meaningful prompts for a target image which can
be used to synthesize diverse images with similar content. We further
illustrate the application of the optimized prompts in evolutionary image
generation and concept removal.

通过使生成的图像内容与用户意图一致的提示，该方法利用扩散模型进行串行优化得到直观的语言提示，从而产生具有相似内容的多样化图像。

提示硬或几乎不提示：用于文本到图像扩散模型的 Prompt 反转

Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image  Diffusion Models

How much private information do text embeddings reveal about the original
text? We investigate the problem of embedding \textit{inversion},
reconstructing the full text represented in dense text embeddings. We frame the
problem as controlled generation: generating text that, when reembedded, is
close to a fixed point in latent space. We find that although a na\"ive model
conditioned on the embedding performs poorly, a multi-step method that
iteratively corrects and re-embeds text is able to recover $92\%$ of
$32\text{-token}$ text inputs exactly. We train our model to decode text
embeddings from two state-of-the-art embedding models, and also show that our
model can recover important personal information (full names) from a dataset of
clinical notes. Our code is available on Github:
\href{this https URL}{github.com/jxmorris12/vec2text}.

文本嵌入可以泄露原始文本的多少私人信息？我们调查了嵌入反演的问题，重构了用密集文本嵌入表示的完整文本。我们将问题作为控制生成的方法来解决：生成文本，当重新嵌入时，接近潜在空间中的固定点。虽然简单的模型在文本嵌入的条件下表现不好，但一个多步骤的方法，通过迭代修正和重新嵌入文本，能够精确恢复 32 个单词输入的 92%。我们训练我们的模型从两种最先进的嵌入模型中解码文本嵌入，并且还表明我们的模型可以从临床记录数据集中恢复重要的个人信息（全名）。