The large-scale visual-language pre-trained model, Contrastive Language-Image
Pre-training (CLIP), has significantly improved image captioning for scenarios
without human-annotated image-caption pairs. Recent advanced CLIP-based image
captioning without human annotations follows a text-only training paradigm,
i.e., reconstructing text from shared embedding space. Nevertheless, these
approaches are limited by the training/inference gap or huge storage
requirements for text embeddings. Given that it is trivial to obtain images in
the real world, we propose CLIP-guided text GAN (CgT-GAN), which incorporates
images into the training process to enable the model to "see" real visual
modality. Particularly, we use adversarial training to teach CgT-GAN to mimic
the phrases of an external text corpus and CLIP-based reward to provide
semantic guidance. The caption generator is jointly rewarded based on the
caption naturalness to human language calculated from the GAN's discriminator
and the semantic guidance reward computed by the CLIP-based reward module. In
addition to the cosine similarity as the semantic guidance reward (i.e.,
CLIP-cos), we further introduce a novel semantic guidance reward called
CLIP-agg, which aligns the generated caption with a weighted text embedding by
attentively aggregating the entire corpus. Experimental results on three
subtasks (ZS-IC, In-UIC and Cross-UIC) show that CgT-GAN outperforms
state-of-the-art methods significantly across all metrics. Code is available at
this https URL

提出了一个 CLIP 引导的文本生成对抗网络 (CgT-GAN) 模型，通过引入图像数据进行训练，结合语义指导奖励 (CLIP-based reward) 和生成文本的自然度奖励来生成与外部语料库相似的文字。在实验证明，CgT-GAN 在各项指标上明显优于现有的最先进方法。