Text-to-image diffusion models such as Stable Diffusion have recently
attracted the interest of many researchers, and inverting the diffusion process
can play an important role in better understanding the generative process and
how to engineer prompts in order to obtain the desired images. To this end, we
introduce the new task of predicting the text prompt given an image generated
by a generative diffusion model. We combine a series of white-box and black-box
models (with and without access to the weights of the diffusion network) to
deal with the proposed task. We propose a novel learning framework comprising
of a joint prompt regression and multi-label vocabulary classification
objective that generates improved prompts. To further improve our method, we
employ a curriculum learning procedure that promotes the learning of
image-prompt pairs with lower labeling noise (i.e. that are better aligned),
and an unsupervised domain-adaptive kernel learning method that uses the
similarities between samples in the source and target domains as extra
features. We conduct experiments on the DiffusionDB data set, predicting text
prompts from images generated by Stable Diffusion. Our novel learning framework
produces excellent results on the aforementioned task, yielding the highest
gains when applied on the white-box model. In addition, we make an interesting
discovery: training a diffusion model on the prompt generation task can make
the model generate images that are much better aligned with the input prompts,
when the model is directly reused for text-to-image generation.

预测生成式扩散模型所生成图像的文本提示，采用联合回归和多标签词汇分类目标的新型学习框架，结合白盒和黑盒模型解决生成文本提示的问题，并通过课程学习和领域自适应核学习方法进一步改进。实验结果表明，该学习框架在预测文本提示任务上取得了出色的结果，尤其在白盒模型上应用时效果最好，并发现在生成文本到图像时将该模型用于回收训练能够使生成的图像与输入提示更好地对齐。