Vision-language (VL) Pre-training (VLP) has shown to well generalize VL
models over a wide range of VL downstream tasks, especially for cross-modal
retrieval. However, it hinges on a huge amount of image-text pairs, which
requires tedious and costly curation. On the contrary, weakly-supervised VLP
(W-VLP) explores means with object tags generated by a pre-trained object
detector (OD) from images. Yet, they still require paired information, i.e.
images and object-level annotations, as supervision to train an OD.
To further reduce the amount of supervision, we propose Prompts-in-The-Loop
(PiTL) that prompts knowledge from large language models (LLMs) to describe
images. Concretely, given a category label of an image, e.g. refinery, the
knowledge, e.g. a refinery could be seen with large storage tanks, pipework,
and ..., extracted by LLMs is used as the language counterpart. The knowledge
supplements, e.g. the common relations among entities most likely appearing in
a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of
14K categories from ImageNet21K with PiTL. Empirically, the VL models
pre-trained with PiTL-generated pairs are strongly favored over other W-VLP
works on image-to-text (I2T) and text-to-image (T2I) retrieval tasks, with less
supervision. The results reveal the effectiveness of PiTL-generated pairs for
VLP.

本文提出了一种利用大型语言模型从图像中生成标签以进行虚实预训练，减少了对昂贵标注数据的需求，并在图片文字检索方面得到显著的效果提升。