Training an image captioner without annotated image-sentence pairs has gained
traction in recent years. Previous approaches can be categorized into two
strategies: crawling sentences from mismatching corpora and aligning them with
the given images as pseudo annotations, or pre-training the captioner using
external image-text pairs. However, the aligning setting seems to reach its
performance limit due to the quality problem of pairs, and pre-training
requires significant computational resources. To address these challenges, we
propose a new strategy ``LPM + retrieval-augmented learning" where the prior
knowledge from large pre-trained models (LPMs) is leveraged as supervision, and
a retrieval process is integrated to further reinforce its effectiveness.
Specifically, we introduce Retrieval-augmented Pseudo Sentence Generation
(RaPSG), which adopts an efficient approach to retrieve highly relevant short
region descriptions from the mismatching corpora and use them to generate a
variety of pseudo sentences with distinct representations as well as high
quality via LPMs. In addition, a fluency filter and a CLIP-guided training
objective are further introduced to facilitate model optimization. Experimental
results demonstrate that our method surpasses the SOTA pre-training model
(Flamingo3B) by achieving a CIDEr score of 78.1 (+5.1) while utilizing only
0.3% of its trainable parameters (1.3B VS 33M). Importantly, our approach
eliminates the need of computationally expensive pre-training processes on
external datasets (e.g., the requirement of 312M image-text pairs for
Flamingo3B). We further show that with a simple extension, the generated pseudo
sentences can be deployed as weak supervision to boost the 1% semi-supervised
image caption benchmark up to 93.4 CIDEr score (+8.9) which showcases the
versatility and effectiveness of our approach.

提出了一种新的策略 “基于 LPM 和检索增强学习” 来训练图像说明器，该策略利用大型预训练模型（LPM）的先验知识作为监督，并通过检索过程来进一步增强其有效性，在不使用昂贵的预训练过程的情况下，超越了目前最先进的预训练模型，并通过使用生成的伪句子作为弱监督来提高 1％半监督图像说明基准的性能。