The rapid evolution of text-to-image diffusion models has opened the door of
generative AI, enabling the translation of textual descriptions into visually
compelling images with remarkable quality. However, a persistent challenge
within this domain is the optimization of prompts to effectively convey
abstract concepts into concrete objects. For example, text encoders can hardly
express "peace", while can easily illustrate olive branches and white doves.
This paper introduces a novel approach named Prompt Optimizer for Abstract
Concepts (POAC) specifically designed to enhance the performance of
text-to-image diffusion models in interpreting and generating images from
abstract concepts. We propose a Prompt Language Model (PLM), which is
initialized from a pre-trained language model, and then fine-tuned with a
curated dataset of abstract concept prompts. The dataset is created with GPT-4
to extend the abstract concept to a scene and concrete objects. Our framework
employs a Reinforcement Learning (RL)-based optimization strategy, focusing on
the alignment between the generated images by a stable diffusion model and
optimized prompts. Through extensive experiments, we demonstrate that our
proposed POAC significantly improves the accuracy and aesthetic quality of
generated images, particularly in the description of abstract concepts and
alignment with optimized prompts. We also present a comprehensive analysis of
our model's performance across diffusion models under different settings,
showcasing its versatility and effectiveness in enhancing abstract concept
representation.

这篇论文介绍了一种名为 POAC 的新方法，旨在增强从抽象概念中解释和生成图像的文本到图像扩散模型的性能。它提出了一个 Prompt Language Model (PLM) 的框架，通过优化的提示来对生成的图像进行对齐，通过大量实验证明，POAC 显著提高了生成图像的准确性和美学质量，尤其是在描述抽象概念和优化提示方面。