Text-to-image generation models have progressed considerably in recent years,
which can now generate impressive realistic images from arbitrary text. Most of
such models are trained on web-scale image-text paired datasets, which may not
be affordable for many researchers. In this paper, we propose a novel method
for pre-training text-to-image generation model on image-only datasets. It
considers a retrieval-then-optimization procedure to synthesize pseudo text
features: for a given image, relevant pseudo text features are first retrieved,
then optimized for better alignment. The low requirement of the proposed method
yields high flexibility and usability: it can be beneficial to a wide range of
settings, including the few-shot, semi-supervised and fully-supervised
learning; it can be applied on different models including generative
adversarial networks (GANs) and diffusion models. Extensive experiments
illustrate the effectiveness of the proposed method. On MS-COCO dataset, our
GAN model obtains Fréchet Inception Distance (FID) of 6.78 which is the new
state-of-the-art (SoTA) of GANs under fully-supervised setting. Our diffusion
model obtains FID of 8.42 and 4.28 on zero-shot and supervised setting
respectively, which are competitive to SoTA diffusion models with a much
smaller model size.

本文提出了一种在仅有图像数据集的情况下预训练文本到图像生成模型的新方法，通过检索和优化过程综合生成伪文本特征，可以灵活应用于各种情境和模型，并在实验中表现出显著的效果，GAN 模型在完全监督的情况下得到了 6.78 的 FID，是 GAN 最新的 SoTA 结果。