Well-designed prompts can guide text-to-image models to generate amazing
images. However, the performant prompts are often model-specific and misaligned
with user input. Instead of laborious human engineering, we propose prompt
adaptation, a general framework that automatically adapts original user input
to model-preferred prompts. Specifically, we first perform supervised
fine-tuning with a pretrained language model on a small collection of manually
engineered prompts. Then we use reinforcement learning to explore better
prompts. We define a reward function that encourages the policy to generate
more aesthetically pleasing images while preserving the original user
intentions. Experimental results on Stable Diffusion show that our method
outperforms manual prompt engineering in terms of both automatic metrics and
human preference ratings. Moreover, reinforcement learning further boosts
performance, especially on out-of-domain prompts. The pretrained checkpoints
are available at this https URL. The demo can be found at
this https URL.

该研究提出了一种文本生成图像的自适应提示适配框架，该框架使用预训练语言模型进行有监督的微调，再使用强化学习探索更好的提示，通过奖励函数鼓励策略生成更具美感的图像，实验结果表明，自适应提示适配方法优于手动人工优化提示工程，预训练检查点已经公开发布。

文本到图像生成的提示优化

Optimizing Prompts for Text-to-Image Generation

Model-agnostic meta-learning (MAML) is a popular method for few-shot learning
but assumes that we have access to the meta-training set. In practice, training
on the meta-training set may not always be an option due to data privacy
concerns, intellectual property issues, or merely lack of computing resources.
In this paper, we consider the novel problem of repurposing pretrained MAML
checkpoints to solve new few-shot classification tasks. Because of the
potential distribution mismatch, the original MAML steps may no longer be
optimal. Therefore we propose an alternative meta-testing procedure and combine
MAML gradient steps with adversarial training and uncertainty-based stepsize
adaptation. Our method outperforms "vanilla" MAML on same-domain and
cross-domains benchmarks using both SGD and Adam optimizers and shows improved
robustness to the choice of base stepsize.

本文提出一种在没有访问元培训练集的情况下，利用预训练 MAML 检查点来解决新的 few-shot 分类任务的方法，通过结合对抗训练和基于不确定性的步长适应，以及对算法进行改进，在 SGD 和 Adam 优化器上，在同一领域和跨领域基准测试中优于 “基准” MAML，并表现出对基础步长的选择具有更好的鲁棒性。