Generative text-to-image models such as Stable Diffusion allow users to
generate images based on a textual description, the prompt. Changing the prompt
is still the primary means for the user to change a generated image as desired.
However, changing the image by reformulating the prompt remains a difficult
process of trial and error, which has led to the emergence of prompt
engineering as a new field of research. We propose and analyze methods to
change the embedding of a prompt directly instead of the prompt text. It allows
for more fine-grained and targeted control that takes into account user
intentions. Our approach treats the generative text-to-image model as a
continuous function and passes gradients between the image space and the prompt
embedding space. By addressing different user interaction problems, we can
apply this idea in three scenarios: (1) Optimization of a metric defined in
image space that could measure, for example, image style. (2) Assistance of
users in creative tasks by enabling them to navigate the image space along a
selection of directions of "near" prompt embeddings. (3) Changing the embedding
of the prompt to include information that the user has seen in a particular
seed but finds difficult to describe in the prompt. Our experiments demonstrate
the feasibility of the described methods.

我们提出并分析了一种直接改变提示嵌入而非提示文本的方法，将生成的文本到图像模型视为一个连续函数，并在图像空间和提示嵌入空间之间传递梯度，通过解决不同的用户交互问题，应用于优化图像空间的度量、在创造性任务中帮助用户导航图像空间以及包含用户在特定种子中看到但难以在提示中描述的信息等三种场景。实验证明了所描述方法的可行性。