Pretrained visual-language models have demonstrated impressive zero-shot
abilities in image captioning, when accompanied by hand-crafted prompts.
Meanwhile, hand-crafted prompts utilize human prior knowledge to guide the
model. However, due to the diversity between different domains, such
hand-crafted prompt that provide invariant prior knowledge may result in mode
collapse for some domains. Some researches attempted to incorporate expert
knowledge and instruction datasets, but the results were costly and led to
hallucinations. In this paper, we propose an unsupervised prompt learning
method to improve Generalization of Image Captioning (GeneIC), which learns a
domain-specific prompt vector for the target domain without requiring annotated
data. GeneIC aligns visual and language modalities with a pre-trained
Contrastive Language-Image Pre-Training (CLIP) model, thus optimizing the
domain-specific prompt vector from two aspects: attribute and semantic
consistency. Specifically, GeneIC first generates attribute-transferred images
with differing attributes, while retaining semantic similarity with original
images. Then, GeneIC uses CLIP to measure the similarity between the images and
the generated sentences. By exploring the variable and invariant features in
the original images and attribute-transferred images, attribute consistency
constrains the attribute change direction of both images and sentences to learn
domain-specific knowledge. The semantic consistency directly measures the
similarity between the generated sentences and images to ensure the accuracy
and comprehensiveness of the generated sentences. Consequently, GeneIC only
optimizes the prompt vectors, which effectively retains the knowledge in the
large model and introduces domain-specific knowledge.

通过无监督的提示学习方法，本文提出了一种改进图像字幕生成通用性的方法，该方法利用预训练的视觉 - 语言模型来学习目标领域的特定提示向量，通过属性一致性和语义一致性优化领域特定提示向量，从而有效地保留了大型模型中的知识并引入了领域特定知识。